Thoughts on designing our API infrastructure

Posted on by

One of the things MALS gets upto, alongside drinking tea and talking about space rockets, is building Application Programming Interfaces (APIs) to allow access to systems we write, or that we’re trying to glue together. We also make heavy use of third parties’ APIs (for example Google Calendars and Microsoft’s Graph API) so we understand the value in being able to programmatically access data held within a system, to extend and embed it within the rest of the University’s IT ecosystem. Until now we’ve made our own APIs on a pretty ad-hoc basis, custom written for each service they support. Looking to the future we foresee more and more APIs being created and potentially getting wider usage, so thought it was about time we planned how we would tackle these.

We’re mostly interested in RESTful APIs, as we find those the easiest and most useful to work with. We have had to use SOAP based APIs in the past and they are rather more painful to deal with, so we don’t intend to build any more of them.  We don’t really care much about the implementation language, although a lot of what we write will probably be in Perl (its our preferred language for dev-ops work here) the API format is by design pretty much language agnostic.

A bit of whiteboard brainstorming threw up a list of things to consider:

  • Authentication/authorization
  • The need for some “public” APIs to provide information from the University to the Open Data community
  • Consistent and extensible URL formatting
  • Utilising suitable scoping and HTTP access methods
  • Using a standard protocol such as OData

Here’s our current thoughts on each of these.

Authentication/authorization

For lots of the APIs  we will need to be able know who is trying to access them and then tailor the data returned to them, to a suitable subset of the data we hold.  In other words, we need to be able to authenticate the API caller and then know what they are permitted to retrieve or alter.

The standard that many RESTful APIs are now using is OAuth2, and we propose to follow suit. We need to determine what OAuth2 flows and grant types we wish to support though.  The use cases we can foresee at the moment are:

  • Trusted server to server APIs, which can use a confidential client credentials flow
  • Web apps on client devices using the authorization code grant flow

When these flows issue access tokens, they should have a maximum lifetime of 3600 seconds (1 hour).  For flows where refresh tokens are available, these can have a longer lifespan – anywhere from 1 hour to 14 days would seem sensible and may be something we should be able to configure on an API by API basis.

In the past, some basic APIs have had access controlled solely using IP based restrictions. We’d like to move away from that, partly to improve security and partly to permit caller mobility within the network, without having to reconfigure the API access controls (for example allowing virtual machines to move between data centres on different VLANs). However we’ll have to support this legacy access control for some time for existing APIs.

Public APIs for Open Data

Open Data is something that organisations have to increasingly interact with, and so we can foresee people coming to ask us to provide APIs for existing data sets and information sources within the University. For some of these it may not be appropriate to ask remote callers to register before accessing the resource, so we need to be able to allow some API end points to have a “default” guest access if no authentication is performed.

URL Formatting

We discussed a number of options for laying out the parts of the URLs pointing at API endpoints. Some options were:

  • /dept/service
  • /dept/group/service
  • /service
  • /service/operation
  • /service/{version}/operation

We gravitated towards the last of these.  We discounted the “dept” and “group” path components because, whilst they might identify the organisational unit responsible for the API at first, changes in management structure are all too common and would soon obsolete these.  We will need to have a database system to record which groups are the “owners” of which APIs anyway, and that information does not need to be exposed to the calling programs.

In the chosen format, “service” is the thing that the API is providing access to (for example “printers” or “interlibraryloans”), the “operation” is what aspect of that service the API is working on (“location”, “request”, etc) and the “{version}” is a monotonically increasing integer version number (of the format v1, v2…). This reflects the structure that companies such as Google and Microsoft are adopting for their RESTful API URLs. Having the version number embedded in the URL allows us to simultaneously support multiple version of an API, which may be useful in migrating to new implementations, extra features, etc.  We thought about having multi-part version numbers, but decided it was cleaner and simpler to have a simple integer for production services.  We can always use “dev” or “beta” for test/dev versions of the API.

The “operation” does not need to include “get”, “update”, “create”, “delete” as part of that path component.  These are given by the HTTP method(s) used to access the API. Behind the scenes the API could be one single program that can handle multiple access methods, or we can use the web server to direct the requests to different scripts based on the requested HTTP method.

HTTP Methods & Accept headers

The HTTP methods that the APIs support determine how data is read, created, updated and removed. We propose the following methods should be supported where appropriate:

  • GET – retrieve resource data
  • POST – create a new resource
  • PUT – update a resource, replacing all existing data with the new data supplied
  • PATCH (or MERGE) – partially update a resource using only specified properties
  • DELETE – remove a resource

As APIs could conceivably return a variety of data formats, we need to be able to allow the client to specify what type it wants.  In the past (in LORLS) we’ve used a query parameter to indicate whether to return XML, JSON or JSONP, but it was decided that we should really use the HTTP Accept headers.  Each API may have to decide on a default format if no suitable Accept headers are passed in, which might be as simple as returning an error.

Cross Origin Resource Sharing

As many of the APIs will be used from Javascript running in a web browser, we may need to support Cross Origin Resource Sharing (CORS).  This means that our APIs will need to support the OPTIONS method as we will have mehods in use other than GET, POST & PUT (for example DELETE and PATCH) and/or may require custom headers in some cases.

OAuth2 Scopes

The use of OAuth2 means that we can also have scopes provided.  We may want to have some generic scopes that all of our APIs can understand (for example whether basic identity data can be read or read/write) as well as scopes specific to a given API.

Standard Protocol Options

Should we use something like OData or Swagger.io to provide a standardized abstract data model for our APIs? They have the advantages that they have a growing eco-system of supporting tools and libraries which would reduce the amount of code each API would require, permit flexible search and retrieval options and can provide metadata about the resource entities and API operations that are available to a client.  The downside is that they will require a lot more supporting infrastructure to be provided, and may be overkill for internal or lightly used APIs.

We may have to investigate if we can provide a minimal, lightweight version of one of these standards so that if it does become useful/important to implement them more fully we already have the basics in place.

Food for thought…………