LORLS

AJAX performance boosts

Just recently I have been looking at tweaks that I can make to improve the performance of CLUMP.  Here are the ones that I have found make a difference.

Set up apache to use gzip to compress things before passing them to the browser.  It doesn’t make much difference on the smaller XML results being, but on the large chunks of XML being returned it reduces the size quite a lot.

Here is an extract of the apache configuration file that we use to compress text, html, javascript, css and xml files before sending them.

# compress text, html, javascript, css, xml:
AddOutputFilterByType DEFLATE text/plain
AddOutputFilterByType DEFLATE text/html
AddOutputFilterByType DEFLATE text/xml
AddOutputFilterByType DEFLATE text/css
AddOutputFilterByType DEFLATE application/xml
AddOutputFilterByType DEFLATE application/xhtml+xml
AddOutputFilterByType DEFLATE application/rss+xml
AddOutputFilterByType DEFLATE application/javascript
AddOutputFilterByType DEFLATE application/x-javascript

Another thing is if you have a lot of outstanding AJAX requests queued up and the user clicks on something which results in those requests no longer being relevant then the browser will still process those requests. Cancelling them will free up the browser to get straight on with the new AJAX requests.

This can be very important on versions 7 and below of Internet Explorer which only allow 2 concurrent connections to a server over http1.1. If the unneeded AJAX requests aren’t cancelled and just left to complete then it can take Internet Explorer a while to clear the queue out only processing 2 requests at a time.

The good news is that Internet Explorer 8 increases the concurrent number of connections to 6, assuming that you have at least a broadband connection speed, which brings it back into align with most other browsers.

Getting closer to production

Production release of LORLS6 (aka LUMP) here at Loughborough is going to happen (hopefully) in the next month or so.  In anticipation of that we’ve been doing some installs and imports of the existing reading lists from LORLS5 so that the librarians can have a play with the system.  Its also allowing us to squish a few bugs and develop a few last minute features.  The big test of course comes when the users start to hit the system.

Searching now added to CLUMP

Last thing I was working on before Christmas was adding searching into CLUMP.  CLUMP’s search function uses LUMP’s FindSuid API to find a list of Module structural units which contain the search term in the selected data element (at the minute the data elements supported are Module Title, Module Code and Academic’s name).

There are two reasons that CLUMP searches for Modules rather than reading lists.  The main reason being, if a module has multiple reading lists it is better to take the user to the module to see all the related reading lists.

The second reason is that all the current data elements that can be searched are related to the module structural units and not the reading list structural units, and it would be a bit convoluted to get a list module structural units and then look up the reading lists for each one.

Christmas comes early

With the snow and ice still covering jolly old Loughborough our thoughts naturally turn to Christmas, or more specifically Christmas presents. Will we get that present we really want or will it be socks yet again! So as you don’t end up with just a pair of socks we thought we’d put a little extra in your stocking: the alpha release of LORLS 6.

Please feel free to download and install this first release of LORLS 6 and let us know what you think of it. However it does come with the following health warning: “This is an alpha release – it’s absolutely NOT recommended for production usage.”

Perl ZOOM issues

We’re getting to the point at Loughborough where we’re considering “going live” early next year with LUMP, replacing the existing LORLSv5 install that we have as our current production reading list system.  As such, we’ve just spun up a new virtual server today, to do a test LUMP install on it.  This machine has a fresh CentOS installation on it, so needs all the Perl modules loaded.  As we use Net::Z3950::ZOOM now, this was one of the modules installed (along with a current YAZ tool chain).

Once we’d got the basic LUMP/CLUMP code base installed on the machine I grabbed the existing LORLS database from the machine it resides on, plus the /usr/local/ReadingLists directory from the LORLSv5 install on there, in order to run the create_structures LUMP importer script.  Which then barfed, complaining that “Net::Z3950::RecordSyntax::USMARC” was a bareword that couldn’t be used with strict subs (LUMP, and LORLS before it, makes use of Perl’s “use strict” feature to sanity check code).

Hmm… odd – this problem hadn’t arisen before, and indeed the error appeared to be in the old LORLSv5 ReadingListsItem.pm module, not any of the LUMP code.  A bit of delving into the modules eventually turned up the solution: the new Net::Z3950::ZOOM doesn’t do backward compatibility too well.  There was a load of code in /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/Net/Z3950.pm that appeared to implement the old pre-ZOOM Net::Z3950 subroutines, but it was all commented out.  I realised that we’d not had this issue before because I’d run the importer on machines that already had LORLSv5 with an older copy of Net::Z3950 on them.

The “solution” was simply to uncomment all the sub routines under the Net::Z3950::RecordSyntax package.  The create_structures script doesn’t actually use any of the LORLSv5 Z3950 stuff anyway, so its not an issue – we just need the old LORLSv5 modules to install so that we can use them to access material in the old database.  I guess this goes to show the problems you can accrue when you reuse a namespace for a non-backwardly compatible API.

Adding and removing group members

Jason and I have been batting code back and forth for the last couple of weeks to provide an API and CLUMP interface to adding and removing people from usergroups.  We’ve gone through several iterations of designs and implementations and now have something that seems to be working OK on our development installation – hopefully soon to appear on the demo sandbox (unless Gary decides that he wants it to work in a different way! 😉 ).

With that more or less done, and some POD documentation tweaks (‘cos I’d been doing a bit too much cutting and pasting between scripts!), I can now go back to dealing with some of the backend management scripts.  The first two of these will be a simple link checker and a cron-able script to (re-)populate the “held by library” information.  Gary has a couple more reports that Library staff have asked for, but they might just require a couple of APIs to be produced to allow existing reporting systems to work (they weren’t part of the distributed LORLS code base – just local Loughborough reports).

Gary has presented the new system to several groups recently with mostly positive feedback.  We’ve just installed a link to this blog (and thus the demo sandbox system) into the live LORLS installation’s managelist script at Loughborough so that more academics will get a heads-up that something new is around the corner.

Is this thing on?

Since we released the LUMP sandbox on to an unsuspecting and innocent world a week or two ago, we’ve noticed that quite a few people have been playing with it.  Which is great – its just what we hoped folk would do.  Have a footle, take it for a spin round the block, kick the tyres, etc, etc.

However there’s been a resounding silence in the feedback front.  Either this means that:

  • folk are so awestruck by its ground breaking novelty that they don’t know what to say,
  • or it just works so nobody has any bug reports or enhancement suggestions,
  • or is so shoddy nobody thinks its worth bothering with,
  • or we released it at a really stoopid time when most University support stafff are (like us) preparing for the onslaught of the hoards.

If anyone who has played with the sandbox does have any feedback we’d love to hear it… even if its just ‘meh’… 🙂

Infrastructure for usergroup management

I’ve been doing a bit of backend infrastructure creation on LUMP today – a set of seven CGI scripts to help with managing usergroup memberships.  These are:

  • Usergroups4SU – provide a list of usergroups that are involved in ACLs for a given structural unit.
  • Members4Usergroup – provide a list of users that a members of a given usergroup.
  • UserGroupMembership – provide a list of usergroups that a user is a member of.
  • Editing/AddUser2Usergroup – add a user to a given usergroup.
  • Editing/RemoveUserFromUsergroup – remove a user from a given usergroup.
  • Editing/AddUsergroup – create a new usergroup.
  • Editing/RemoveUsergroup – remove an empty usergroup (ie one with no members).

Jason is going to take these in the next week and see if they are sufficient to allow CLUMP to provide a nice graphical interface for staff, librarians and administrators to add and remove users from user groups.  We’ve already got the Editing/EditACLs API CGI script though, so Jason can already tinker with the ability of usergroups to view and/or edit structural units.  We might find that we need some other API calls for admin usage, but this little lots should give us quite a bit of functionality.

Librarian Centre CGIs – departmental breakdowns

Today was a bit of a change of pace in LUMP development – it was time to whizz up a backroom script for the librarians, allowing them to see how many modules, reading lists and different types of items were used by departments in a particular year.  On the whole this was an “old skool” CGI script that doesn’t use the LUMP XML API – just straight calls to the underlying Perl modules that wrap round the LUMP database.  This is both quick to run and quick to code, and for a simple statistics script is fine (no need for flashy JavaScript for those really).  Gary is pondering other statistics scripts we might want to produce, both derived from the existing LORLS ones and also some new ideas.  He’s going to chat to some of our librarians about those first though – no point wasting time writing code to produce information that nobody is interested in!

Installer hits a sticky patch

You know I loved CPAN until I started writing the LUMP installer but now I’m not so keen!

It looks like our cunning plan of allowing people to install the required modules to run LUMP from CPAN using point-n-dribble web front end might have come unstuck for a couple of reasons.  The major one is having Perl modules that use XS to rely on C code/libraries/etc that might not actually be installed on the target machine.  For example we need to use the ZOOM Perl module for Z39.50 access.  This in turn makes use of the Yaz toolkit, which you either need to build from source or get an RPM for… and there aren’t any RPMs in the standard CentOS repos (you can pull CentOS RPMs from Index Data’s web site but not get them via yum).   Similar requirements for Expat, although at least that has yum’ed packages in CentOS.  I not noticed either of these because on my Debian Lenny based workstation there were already handy packages in place and the Perl modules that I’d been “needing” were pure Perl.   I only stumbled across the issue when Jason created a “sandbox” virtual machine running CentOS and I tried to run the installer on there.  Even if I could guess the package manager on use on the machine (yum, apt, etc) I’d need to ask the Admin for the system root password and then have to deal with the out of repo code such as Yaz.  At this point the law of diminishing returns kicked in and we decided we’d wasted enough time trying to make a user friendly installer and it wasn’t going to be easy to get round these problems.

Still, all is not lost: at least the installer code can spot which modules are missing so that you can install them by hand.  In fact I’ve left the code into install private CPAN modules – we might as well do as much as possible for the prospective Admin even if we do then need to tell them to break out the C compiler and build some support software.  And of course the installer can still do everything else once the required Perl modules are in place – create the DB, import test data, and retrieve and configure the various scripts required.

I also accidentally stumbled across another problem today: DBD::MySQL doesn’t allow column filters in the column_info() DBI method.  I was using that to check if the LUMP database schema on the target machine, if present, was out of date compared to the schema embedded in the installer.  This is a bit of a pain but relatively simple to code round.  The DBI manual page implies that if a DBD module can’t do filtering you might have to handle it yourself: DBD::MySQL just barfs with an error message so I’ve had to embed the call in eval() checks just to be on the safe side.

I’ve also just noticed that I’ve hard coded the Kerberos TGT realm!  Oops – we’ll need to change that, especially as we’re just moving ActiveDirectory trees here at Loughborough anyway!  We’ll probably also need to add in an override ability for folk with no AD or other Kerberized authentication infrastructure – that shouldn’t take too much work as its just a little tweak inside the LUMP.pm password validation method.  Hopefully in a week or so we’ll have little demo system on our sandbox VM that we can then let people play with – a bit like the demo system that we already run for LORLS v5 (our “groundhog day” install, that gets blown away every morning in the wee hours and replaced with a fresh copy… in fact I used the LORLSv5 to LUMP importer to convert that database into our new format for this new sandbox machine as another useful test).

Go to Top