Jon Knight

Whilst writing a little dev script to nullify Moodle IDs for Jason, I realised that things could be a bit slow sometimes searching for SUs when the system had a large number of reading lists in it. Once again I’d made the school boy error of forgetting to create indexes in the DB schema for the foreign keys in the various tables. To that end the schema now has indexes on any ID fields in any tables, plus a couple of obvious ones we might want to search on. We’ll probably have to add more in the future, especially on large tables (structural_unit, data_element, access_control_list, user, usergroup for example).

Also been trying to track down uninitialised variables in the build_structures LORLSv4 import script – it works at the moment but looks messy and I’d rather have properly error checking code.

Had to slightly change the output XML when using GetStructuralUnit CGI script API. Originally if asked for HTML the code would just generate the HTML (or whatever format) for each child to the required depth level and just concatenate it. However this meant that higher levels couldn’t make formatting decisions based on details of lower levels – for example reading list SUs need to know when to turn off/on list tags in HTML depend on if children are notes or book/journal/article/etc SUs. The tweak was to always have the child return a structure:

$child->{su}->{render} = formatted output $child->{su}->{id} = child SU's IDThe child SU's ID field lets the parent's code make more intelligent rendering decisions.

We realised that we need to allow member of the Admin group to be able to act as other users. For example Moodle won’t know the user’s AD log in information but we still want LUMP to allow the Moodle logged in user to create, edit, delete and view LUMP objects as themselves (rather than just a default Moodle user). Thus we need to allow admin users to login with their own credientials (using one of the supported authentication mechanisms) and then switch to act as another user. In the web API this is achieved by passing an “act_as” CGI parameter filled in with the username that the script should appear to be running as. Currently only members of the Sysadmin group will have this power (which will need to include the Moodle block as “trusted” code).

We want LUMP to have a well separated client-server architecture, so the we can deploy several different types of front end and also so that third party developers can implement their own front (and back!) ends independently. We’ve pretty much decided on using XML to return the results (as most client programming environments have some sort of XML handling these days, and if the worst comes to the worst there are network XSLT transformation services that can turn the XML into renderable HTML). However we needed to decide how to send requests from the client to the server in the first place.

The initial plan was to have clients lovingly hand craft XML documents for the request and then send these as a single parameter to a CGI script. This XML document would contain all the authentication, version, metadata for the LUMP protocol, as well as the method being requested and any parameters required. It would work OK, but meant that the clients needed to generate an XML document, even for a relatively simple request.

Next we considered the Simple Object Access Protocol (SOAP). SOAP basically does the XML packaging of parameters and results in a standardised manner that is supposed to allow different programming languages on different platforms to remotely access objects and methods as though the were part of the local application. That’s the theory at least. SOAP works better when coupled with the Web Services Description Language (WSDL) but there’s also where things start to come unravelled.

Firstly WSDL appears to have real difficulties in handling dynamic data structures. Most of the examples of WSDL on the web pass a string or two into a routine and then get out a single string, floating point or integer. But we’d like to return far more complex data structures such as nested hash of hashes, arrays of hashes, etc, etc. We could describe those in terms of an XML DTD for our hand crafted responses but not easily in WSDL for use with SOAP. WSDL also imposes an extra transfer over head on clients and/or requires that the client cache the WSDL file (which makes developing a pain, where the WSDL would have to be in constant flux normally and thus uncachable).

There were also some practical problems with SOAP. The main development of the LUMP server is likely to be in Perl and, whilst there is a SOAP Perl module available, its nowhere near as well developed as many of the other Perl modules available. Secondly we found that sometimes we could get SOAP to work with, say, Perl, but then not work with PHP (PHP being the other major target language, this time for the Moodle client block). Fixing access for the PHP client could then cause problems for the Perl client. Why things some times worked and sometimes didn’t wasn’t particularly clear and therefore this didn’t show SOAP to be particular platform independent. And debugging the SOAP transactions was also very hard – even with debugging turned on in the SOAP libraries the interactions can be very complicated and you end up having to sit with the SOAP specification trying to work out which bit of the libraries that were supposed to make life easy for you were actually making it much harder!

So we opted for a third choice, and one we were already quite familiar with: REpresentational State Transfer (REST). APIs that use REST (so called “RESTful” APIs) make use of the existing web infrastructure such as HTTP, SSL, web server authentication techologies and CGI scripts. A RESTful API uses a base URI pointing at some sort of program (such as a CGI script or mod_perl instance) combined with URI query parameters to specify the operation required, user authentication information, request details, etc. Many programming environments have built in support for accessing URIs and so these RESTful APIs can reuse existing technologies without needing new, complex and potentially buggy libraries. Generating the URIs is relatively simple and straight forward in most languages.

The results of a REST transaction can be anything you like, so there was not reason why it couldn’t be the sort of XML result documents that we looked at originally. This gave the result some platform neutrality, whilst still allowing us to return what were effectively dynamic data structures. It also means that you can easily do funky things with XSLT still if you want to. The clients can process XML returned fairly easily these days (as evidenced by the AJAX mash ups out there) but they wouldn’t need to go through the overhead of generating a new XML document for the outgoing request itself (unless it was required for a complex parameter perhaps – that’s flexibility for you!).

RESTful APIs also have a big conceptual win over a SOAP based API: the interface gets real URLs for each request. This means that not only get you retrieve object details using a suitably crafted URI, but also create, update and delete them (assuming you have rights to obviously). Having such URIs available means that the other parts of the W3C’s web standards development dovetail in nicely with RESTful APIs: things like RDF and XPointer only really become useful if they can reference resources using URIs.

Having chosen (for the moment!) REST as API mechanism there appeared to be one decision still outstanding. That is where to draw the line between the “base” URI and the parameters. One option is to have a single base URI for all operations and then a parameter (amongst others) that describes the operation we required. For example a FindSuid call might look something like:

http://example.com/cgi-bin/LUMP?operation=findsuid&what=Module&aspect=Module%20Code&value=08LBA100

The other option is to have different base URIs for each of the “methods” that the API has. For example the base URI for the above operation might be:

http://example.com/cgi-bin/LUMP/FindSuid

and then we’d add parameters to the end of that, giving a URI of:

http://example.com/cgi-bin/LUMP/FindSuid?what=Module&aspect=Module%20Code&value=08LBA100

There’s not really much in it as far as we can see at the moment. On the one hand having a single base URI might mean that the client’s configuration is very, very slightly simpler. On the flip side having separate URIs might translate to having lots of small, simple CGI scripts, as opposed to a single monolithic script. The small scripts might or might not be easier to maintain, depending on how much code reuse and modularisation we can come up with. Also if we use something like mod_perl with the scripts to speed things up, the smaller units will hog less memory in the web server, take less to get running and can be updated with less impact on a running service.

Either of these would appear to be RESTful as they both result in workable URIs. Luckily we can have our cake and eat it in this case. If we opt for individual URIs for each method then we can take advantage of the back end benefits, but also then use existing web technologies to make them appear as a single monolithic interface with a single base URI and an “operation” parameter if we wish to. This could either be through a wrapper CGI script or one of the web server URL rewriting technologies (though Apache may have issues with rewriting parameters).