[cpg1.5.x]: Updates on a next releases of CpmFetch

[vuud]

After many false starts of carving out time to get this done - this time I wanted to make sure I got some traction and some progress before I spoke up again.  Since I do, I wanted to have somewhere I could post progress as to what I was doing and be a little more accountable on a daily basis.

I am also now on the dev team for CPG, so I feel a bit more compelled to contribute.  I wonder if this is what GauGau had in mind... hmm...

First some bad news:  I don't plan any more releases supporting versions of Coppermine before the upcoming version 1.5, and the newer version will have some PHP version requirements that exceed that of Coppermine Photo Gallery.  I am sure it will alienate someone out there, but there is no way the new code will run on PHP4.  Support for PHP4 was years ago, and if your host is still only supporting that, time to rethink things.  If you have code that is in 4... it is probably already vulnerable to exploits, etc...

So there has to be some good news in here right?

Yes there is... there is some really really cool stuff for everyone, regardless of your PHP skill, or what you are using this for.

Basically it is going to be waaaay more versatile for all concerned moving forward.  Since everything I write here is taking away from coding time, I am going to keep updating this topic - hopefully each night after I get some code done, if not every other night depending on the time of week.

So the next post will be about the architecture of the new code and what the benefits are:

[vuud]

CpmFetch will now be in server parts

Parts?  Yes.  GauGau pushed me toward implementing CpmFetch as a plugin under 1.5's architecture.  Plugins are a really really cool thing that has been in CPG for awhile - but I've not really played with them until now.

Now, doing it as a plugin means that installation will be a snap (similar to adding a theme), and I get to use a lot of code directly from the Coppermine Photo Gallery as is... I don't have to duplicate all the permission checking, etc like I had to do before on my own.

The downside is that it becomes problematic to get the data back into your pages easily.  Also - each time you made a request it has to start up a big chunk of CPG to execute your call.  If you put three different displays on your page at the same time, it would have to start CPG three times.

The solution:

CpmFetchService:RPC - this will be the plugin that sites on the server.   It will accept requests in a format called JSON-RPC.  To the layperson, this means that the client web site can bundle up as many requests as they want and submit them to the plugin.  The plugin will process them all individually and send back the results.  One call and start up for as many things as you want.  

For the hardcode coder out there - the JSON-RPC layout is very very easy and I plan on documenting the available calls very well.  This means that this component can be talked to by anyone out there writing in PHP, ASP, PERL, HTML & JS, etc...  You won't need to deal with a client side cpmfetch library at all.

For the layperson - I play on recreating the cpmfetch code similar to how it is now - except you will:

• Have a very easy configuration - where is the http://www.yourgallery.com site
• No limitations on crossing servers or different sites
• Configuration and control right in the CPG gallery for most options
• It will talk to CPG, not directly to the database

My thoughts at this time are to develop a basic PHP library to handling making the request for people who don't want to deal with crafting JSON-RPC, and a similar offering in pure javascript.  Then adding on to that.

Also, since the CpmFetchService is going to be coded around getting data, I will be expanding that into other areas like:

CpmFetchService:RSS
etc...  but those are farther down the road.

I am most excited to see what people do with just getting at the data directly in combination with all the cool js / jquery type effects out there...


What did I get done tonight and up till now?
Well, these are not meant for everyone to understand - but more to keep me moving, so some of you will understand the progress, some may not.  

• All code is being done with unit tests
• All code is being PHPDoc'd
• All code is being done in an OO fashion
• I am making sure the code is beautiful - not slapped together

• Created a test db and gallery in 1.5 to test against
• Completed a proof of concept test with the plugins
• Completed a proof of concept with JSON-RPC calls
• Designed an JSON-RPC implementation in PHP to handle it on both ends
• Coded a unit test and Request object
• Coded a unit test and most of the Envelope object (to put multiple requests into it)
• Migrated a Response / Error Response object from the proof of concept into better code

See you tomorrow.

EDIT: Went back and completed the envelope object including implementing iteration across it.  I should be able to start firing test requests at the CpmFetchService:RPC in the next day or two.

[vuud]

Minor progress tonight

I spent most of tonight completing the envelope code for the server side of things (handling the requests that get sent in). Finished up more of the unit tests and resolved a few bugs that came up.

I refactored the response object and move some base functionality it shares with the requests into a base class.  Probably not worth it for what I have, but moving forward it could come in handy.

Right now I have about two more functions to code and then the communications layer is done.  Now I can talk to Coppermine Photo Gallery, I just need to set up what I want to say, and what it wants to hear. 

I am looking forward to that part, as that is when I start to see tangible results coming back.  Pictures and stuff.

I really want to do it now... I really do, but the lack of sleep the past few nights is catching up with me.  So, over the weekend hopefully I can get some late nights in (when I work best).

Later

[vuud]

Zoinks

Tonight I wrapped up the code for the communications layer.  What normally would have taken 20 minutes ended up taking all night.  I decided that I did not like the way the naming was going on the functions and I really wanted to use a fluent interface where I could.  What does that mean?

Basically on items that do not give you back something, you can make the code a bit neater:

So the conventional way is:

Code Select Expand


$req = new Cpmfetch_Jsonrpc_Request();
$req->setMethodName('testmethodname');
$req->setId(1);
$req->addParameter('name1','value1');
$req->addParameter('name2','value2');


With a fluent interface it looks more like:

Code Select Expand


$req = $this->Cpmfetch_Jsonrpc_Request();
$req  ->setMethodName('testmethodname')
->setId(1)
->addParameter('name1','value1')
->addParameter('name2','value2');


Okay, so it does not buy you that much, but it does make for more readable code.

I did also put in a lot more code to prevent someone from breaking the JSON-RPC spec.  I also have a test suite that runs 50 or so tests against the code everytime I make a change anywhere.  Found a bunch of bugs this way.  For anyone doing PHP, I recommend PHPUnit.  Its lots of fun.

I did realize that I myself have broken part of it - that I need to work out before moving on.

* If you do not set an ID, the server should treat it as a notification.  My system will helpfully assign an ID if you don't.  Not that bad, except on the server end it will probably try to send back a result, which it should not on a notification.

We will see later on how this gets fixed.  I am going to get me a snack and maybe work some more on the cpg plugin (finally)

Vuud

[vuud]

Whahahahahahaaa - It's ALIVE!

I spent some time today on implementing the communications layer and connecting it from the plug in on the server.

I am excited enough about this to share a screenshot and code from what I tested.  This is the cpmfetch equivalent of Bell's first phone call.  Did not accomplish much, but worked. (screenshot attached)

So the code below is typically what the cpmfetch client program would use to contact the server.  Of course, you could use this straight up if you wanted it.  I figure there is going to be three levels of users:

1) Convenience users - use a cpmfetch like interface to say "give me 10 random pictures"
2) PHP Coders - will use the cpmfetch json-rpc library to get data and format it themselves
3) Self-Implementors - who craft their own JSON-RPC requests and fire them off against the plugin.

So this code reflects more of the #2 above...

Code Select Expand



$env = new Cpmfetch_Jsonrpc_Envelope();

$cjr = new Cpmfetch_Jsonrpc_Request();
$cjr->setMethodName('getCoppermineInformation');
$cjr->setId("Info");

$env->addRequest($cjr);
$env->postRequest('http://cpmfetch3.localhost.com/Cpg15/?file=cpmfetch/jsonrpc');

$resp = $env->getResponse("Info");
$results = $resp->getResult();
echo "Gallery name:   " . $results['gallery_name'] . "\n";
echo "Description:    " .  $results['gallery_description'] . "\n";




Okay, a quick run down of the code and what it is doing...

Code Select Expand


$env = new Cpmfetch_Jsonrpc_Envelope();


The envelope is what holds and handles all the requests, responses, etc...

Code Select Expand


$cjr = new Cpmfetch_Jsonrpc_Request();
$cjr->setMethodName('getCoppermineInformation');
      ->setId("Info");
$env->addRequest($cjr);


Above we are creating a new request.  We are then telling the server that we want to run the getCoppermineInformation routine in the plugin.  Importantly we assign it an ID of Info, so we know where to get it out later on. 

Note:  You can add as many requests as you want (and you should) to a single envelope.  Multiple envelopes will result in a performance hit, as each will make its own connection to the server.

Note:  The requests already can handle parameters being sent, I just did not make an example that uses it yet.

Code Select Expand


$env->postRequest('http://cpmfetch3.localhost.com/Cpg15/?file=cpmfetch/jsonrpc');


This actually makes the entire JSON-RPC requests, creates a POST connection to the URL, sends the data, gets the responses back, decodes them, and adds them to your envelope.  A ton of stuff goes on in there.

Code Select Expand


$resp = $env->getResponse("Info");
$results = $resp->getResult();
echo "Gallery name:   " . $results['gallery_name'] . "\n";
echo "Description:    " .  $results['gallery_description'] . "\n";


Remember how I said giving the ID was important?  Here we get the response out, and the results.  From there it is just a big array of data.

With one request it does not make much sense, but when you stick a few in there you will need it

[vuud]

Design of server side extendibility for intermediate to advanced users:

Okay, so I have talked about how you can just make calls to the CPG server through PHP and stuff.  Well to make that happen the CpmFetch Service needs to have the function created in it first.  So what happens when you have something new you want to try but don't want to develop your own plugin or can't stomach messing up the impeccably written Cpmfetch Service?  Well, you add a custom library to the mix and add your own function.

Lets take a simple example, where we multiply two numbers.  (this code really works, its not a prototype)

First here is the code you would use from the client side (your web site or app):

Code Select Expand


$env = new Cpmfetch_Jsonrpc_Envelope();
$cjr = new Cpmfetch_Jsonrpc_Request();
$cjr->setMethodName('multiplication');
      ->addParameter("number1", "100"),
      ->addParameter("number2", "A"),
      ->setId("Info");

$env->addRequest($cjr);
$env->postRequest('http://cpmfetch3.localhost.com/Cpg15/?file=cpmfetch/jsonrpc');

$resp = $env->getResponse("Info");
echo $resp->getResult();



Okay, so we are taking two numbers, and multiplying them.  Here is the function you would write on the server in the user library for the cpmfetchservice.

Code Select Expand


public static function method_multiplication( $parameters = NULL)
{
$ap = array();
$ap['number1'] = array('validate' => 'numeric', 'required' => TRUE);
$ap['number2'] = array('validate' => 'numeric', 'required' => TRUE);

$return = new Cpmfetchplugin_Library_Result();

if (Cpmfetchplugin_Library_Base::validateParameters($return, $parameters, $ap)) {

                       // Your code would go here... replacing this line that does the multiply
$return->setResult($parameters['number1'] * $parameters['number2']);
} else {
// if you want to do any more with the invalid parameters you can here, otherwise
                        // the error message is fed back as an error to the calling client program as a
                        // JSON-RPC error message for this call.
}

return $return;
}
   

One thing you will notice is that all the "heavy lifting" is done for you.  The system takes the request, handles it, and passes the data right to your function.  In fact lines 3-5 set up data checking so you can easily sanitize (or at least verify your input).

The value checking so far supports some basic PHP types and looks for missing required parameters.  I will be adding in some SQL injection protection soon also.

The people that just use Cpmfetch to grab an image or so are probably like "huh?" right now.  Somewhere there are PHP users that tinker in CPG and are psyched as I am about this.  I really have to believe that

So what's up next?  I am going to start looking at returning actual picture data.  Since CPG already has tons of functions inside it - I am really just digging those out and making the equivalent of the above    The CpmFetch I had to recreate all the SQL for each call. 


I am going to create a new thread here to take questions about the usage of this, etc... 

[vuud]

Tonight I was able to code up a routine to get back random photos and some associated data (owner name, album name, etc).  Nice, but I am not happy with how it is going into the cpg code.  May have to code more than I thought on that side.  Oh well, it still won't be as bad as the original coding in cpmfetch.

Yay.   Then I accidentally overwrote everything I had done the past few days with the 5.2.3 release candidate (OSX and Linux/MC copies of directories work very very differently).

Luckily time machine had a backup from 30 minutes before, so I actually lost nothing.  If you have a mac and you are not using time machine, you really need to use it.  Free + the cost of a cheap external drive.

[vuud]

I ended up getting sucked into some work issues that took all night to debug.  On the other hand, I also took apart my whole testbed installing the new release candidate 5.2.3.

The new default theme is pretty nice I have to say - a pleasant surprise.

I am going to just touch up some doc's tonight and pick up again where I left off tomorrow.

[vuud]

I've taken a break from coding to get the recent beta running of 1.5 - and I've taken this time to bring together all my thoughts on how this all comes together.  At this point I want to make sure the foundation I have put down is the right one - I don't want to have to rework it all.

Anyway, I have put together a graphic illustrating how the all comes together, from the client side to the CPG side.  Any questions or suggestions can be made in the other thread for open discussion of this version.

In the meantime, see the attached graphic.

Thanks!

[vuud]

Well, a particularly wasted day.  I decided this morning to rename everything, then had some coffee and came to my senses.  Spent the next two hours refactoring my changes back.  Normally I wait until I have a stable base before committing it into my SVN repository, maybe I have to rethink that.

Anyway, did complete the coding up of a user library... so you can write your own functions pretty easily (basic PHP skills needed, and an understanding of what you are looking for) and include them without interfering with the CpmfetchService code base at all.  This will also make support easier, as you can just mail me the file and I can drop it in place.

Started also neating up the plugin code base, fixing names etc.  I hope to have something I can show the other dev's later this week... I may not get too much done next week, so the review break could be a good thing.

Found a fixed some bugs in and accounted for some php oddities...

Things are going well, still lots more to do - but I am pleased with where I am now and heading.

If you have questions about this, see the parallel thread and post one there - if there is any feedback post there also.  thanks!

[vuud]

The sage continues...

For that matter, whatever happened to the band "Saga"?

Anyway...  I have decided to implement some more structure into the JSON-RPC setup.  Typically each batch of requests should be able to be handled in any order and not be dependent on each other.  I've decided to implement special calls that would get processed before or after the rest of the calls.

Before

Auth - the can potentially allow you to authenticate with the gallery to get access to more stuff.  In other cases, it may just be a present password on the plugin to regulate access.  Not sure, but it would be grabbed out and handled first.

Settings - this would change the default settings for the duration of the plugins execution.


After

AddUserInfo
AddAlbumInfo
AddCategoryInfo

Basically the thought on the last three is that bundling in all the user and album info on each row returned will start to eat up a lot of bandwidth.  I am thinking that I can have the library keep track of all the user id's, albums, and categories it sees - and return the info all at once.  So if you had a 1000 photos returned, all belonging to 2 users, you would only get 2 user records back - not the user data added onto each photo returned.  It would be up to the client to cross reference it.  By separating it from the method asking for the pictures, it can apply to all the images returned across all the methods in that request.

In layman's terms, it will be faster.

[phill104]

Keep up the good work. I find it very interesting to see the development.

[vuud]

Keep up the good work. I find it very interesting to see the development.

Thanks... if you have any questions on it, post to the QA thread.  Seems to be nothing happening in there

[vuud]

I've spent some time trying to decide how to integrate the CPG database stuff into my code...  The problem that I am running into with the CPG code is that it tends to be written (at least in some cases) around a particular task... so generating a select box, etc...  I can't just get the data back.

So... I am going to start building out a database layer that lets me go after what I need while taking advantage of the CPG code I can use.  This will also let me test a bit easier. 

What I started last night is a set of objects to get the data back by passing in the DB connection, the existing forbidden stuff, maybe a user and just asking it for pure data back.  We will see how it goes, and it will have to expand as I need new stuff - I am not going to write it to cover every table right away.  The good news is that I can decouple it from CPG and test it directly instead of relying on a web service in to execute it.  I love OO.

But anyway...  I want to make some progress a few nights this week coming up.  I should know pretty quick how good/bad this will work out for me.

[vuud]

So database work continues, slowly.

I had a week off and spent a ton of time framing up my basement.  In the process I managed to hurt my lower back enough that I have been told to either be lying down or standing up...  I can sit for awhile, but then it really starts hurting.  So not much dev happening till we get this sorted out.  X-Rays in two days, so we will know more then

On the bright side, framing is done

[vuud]

So I have been having back pain and generally been laid up since I worked on my basement.  I am trying to get back into the DB work I started, but its been a bit and I completely forgot my train of thought in the development...

So not out of the game, just a rain delay.

Maybe food and a nap would help.

Sorry

[vuud]

Well, the back pain is gone (mostly) which is good.  But we are into gardening season, which means getting the beds ready.  But enough about my pain and suffering.

I've cracked open the code and decided to take a much simpler approach to accessing the CPG database.  My original plan was to build a system that could be used in the actual project if it worked to replace what is happening now.   That, is turning out to be a bit more than I can get done with my time constraints.  I can look at it later on, but for now - it will simply delay CpmFetch from being completed. 

So my plan is to replicate the existing DB functions as I need them, and taking out the parts I dont need.  For instance, there is a function in the main CPG code that does exactly what I want *except* it will only return a select box of the results.  I will basically re-write it to do the same, but return data (which should then be decorated with a select box if someone needed too).

Also, I am putting this up on bitbucket... so once it's released anyone can have access to my source control, tests, etc...  Not for the faint of heart - but at least its out there in case I get hit by a runaway moped.  Reorganizing the code as a break from gardening.

Talk to you all soon!

(When I say gardening I mean to say I had 3 x 4' x 54' x 3' deep trenches dug in my yard.  I am hand screening all the dirt and mixing in compost.   With luck I will never have to go through this again.).