REST JSON Query

Goal: JSON request for topics that start with a given stem

This tutorial will illustrate how to add a new REST interface to spnet, in this case a simple search that will return a JSON list of topics that begin with a specified stem.

Getting Started

OK, where do we start? Spnet follows REST design principles. So let's translate our goal into a REST interface. Generally speaking this means a URI of the form /COLLECTION/..., where COLLECTION is the name of the collection and ... is either an ID (for a specific item in the collection) or nothing if we're doing a search of the collection. So in this case our URI will just be /topics, and our request will be a GET with a search parameter like stem=bayes and Accept header of application/json.

Next, how do we implement that interface in the spnet code? The spnet REST application "tree" of such REST URI interfaces is defined in spnet/apptree.py, in two parts: we add custom behaviors for a given collection by subclassing rest.Collection and adding custom methods like _GET(), _POST() or _search(); and we link them to URIs in get_collections(). For example, here is the current code for the /topics URI:

topics = rest.Collection('topic', core.SIG, templateEnv, templateDir,
                         gplusClientID=gplusClientID)

A few notes:

• /topics is just using a generic rest.Collection with no custom methods. So we'll create a new subclass and add a _search() method.
• its various arguments specify its variable name (topic) to be passed to templates, the data class (core.SIG) to be used to instantiate a given topic ID, template loading information, and extra keyword arguments to be passed to templates (in this case just gplusClientID).

Implementing a search

We can simply use a mongoDB regular expression search to find all topics that start with the specified stem. In pymongo this is just a query of the form coll.find({'fieldname':{'$regex': '^bayes'}}), e.g. to find records in collection coll whose field fieldname starts with "bayes". Let's test that directly in a Python session (e.g. start python -i web.py)

>>> list(core.SIG.find({'_id': {'$regex': '^b'}}))
[u'banachSpaces', u'base', u'bayesian', u'bigdata', u'billiards', u'black_holes', u'blackholeentropy', u'braids', u'breakthrough', u'bredonHomology', u'burst', u'byAuthor']

Works great! So we just subclass rest.Collection and add a _search() method that does just that, taking advantage of the fact that any rest.Collection knows what data class it's working with, as its klass attribute:

class TopicCollection(rest.Collection):
    def _search(self, stem): # return list of topics beginning with stem
        if not stem:
            return []
        return list(self.klass.find({'_id': {'$regex': '^' + stem}}))

We switch the topics link to use our new subclass, by changing that line to read:

topics = TopicCollection('topic', core.SIG, templateEnv, templateDir,
                         gplusClientID=gplusClientID)

Restart your Python session, and check that this works as expected, noting that spnet/web.py creates its Server object as s, so we can call our URI interface just so:

>>> s.topics._search('b')
[u'banachSpaces', u'base', u'bayesian', u'bigdata', u'billiards', u'black_holes', u'blackholeentropy', u'braids', u'breakthrough', u'bredonHomology', u'burst', u'byAuthor']

Excellent.

Providing a JSON interface

There's just one thing left to do: provide a method that will handle JSON requests. Note that REST insists on a separation between verbs (actions like GET, POST, DELETE) and the desired format for returning data (e.g. HTML, JSON, specified by the HTTP Accept header). rest.Collection follows that principle, by looking for a method whose name is of the form VERB_FORMAT(), where VERB is something like get, post, search, and FORMAT is something like html or json. So we need to implement a search_json() method that returns a string representation of our data in JSON format. That's easy; now our code looks like:

class TopicCollection(rest.Collection):
    def _search(self, stem): # return list of topics beginning with stem
        if not stem:
            return []
        return list(self.klass.find({'_id': {'$regex': '^' + stem}}))
    def search_json(self, data, **kwargs):
        return json.dumps(data)

The only fiddly detail here is the requirement for a generic **kwargs argument. This is because rest.Collection sends the same keyword arguments to search_json() as it sent to _search() (in addition to the data returned by _search()). If search_json() doesn't accept those keyword arguments, that will cause a server error. In this case we don't have any need to use those arguments, but our method must accept them. Save your code and restart the spnet server (using the new code).

Now let's fire up another Python session and use the handy requests library to check that our URI is working as expected:

>>> import requests
>>> r = requests.get('http://localhost:8000/topics', params=dict(stem='b'), headers=dict(Accept='application/json'))
>>> r.status_code
200
>>> r.json()
[u'banachSpaces', u'base', u'bayesian', u'bigdata', u'billiards', u'black_holes', u'blackholeentropy', u'braids', u'breakthrough', u'bredonHomology', u'burst', u'byAuthor']

Great! We're done.