[nolan@nprescott.com] $>  cat blog archive feed

Twisted's Klein

2018-10-06

In doing some greenfield development at work recently I've been digging into Twisted and the Klein "micro-framework". I'm really coming to appreciate some aspects of the projects as a result.

The project I'm working on is a stand-alone utility to, primarily, surface information via a series of web-views in both a human-readable and machine-readable format and secondarily, to subscribe to a set of updates via message queue and write them to a database.

Doing Two Things At Once

I feel like this is an under-appreciated aspect of Twisted, I know I only discovered it in earnest when I first made my system monitoring service, but the application and service architecture makes it very easy to combine discrete components in a easily separable way. In this instance I wanted to an HTTP server to surface both kinds of web-views, and an AMQP client to interface with the message queue. In my particular case, the both the HTTP server and the AMQP client relied on access to the same database. It was a simple thing to establish an asynchronous connection pool with Twisted adbapi abstracting both interfaces behind a single class. It was a cinch to get all three pieces, shared database access, client, and server running under the same process.

Rough Edges

As part of the dual-format presentation (human-readable and machine-readable), I wanted to minimize the duplication without having to go "full single page application". Part of what lead me to choose Klein was the Plating module, which ties in a basic templating system along with an automatic JSON interface for each route. I'll be the first to admit that Klein isn't widely used, there aren't many answers to be found on StackOverflow and no one at work had used it before. This would be a pretty big downside but for the fact that it is very easy to just read the source and answer questions on my own.

As a particular example, the plating mechanism is pretty rudimentary in handling nested data. I had a data-set that looked something like the following:

'groups': {
  'group 1': ['foo', 'bar'],
  'group 2': ['baz', 'qux', 'asdf']
  ...
}

And wanted to surface it into an HTML view like the following:

The automatic handling of iterable data first "flattens" things, so there wasn't a good way to display the above with just the render='groups:list' format that the documentation describes. It is a pretty simple matter to instead design a function to handle the data myself:

def grouper(data_dictionary):
    return tags.ul(
        [tags.li(group)(
            tags.ul(
                [tags.li(item) for item in data_dictionary[group]]))
         for group in data_dictionary])

The above is results in a data structure like this (with some whitespace added for clarity):

Tag('ul', children=[
    [Tag('li', children=['group 1', 
        Tag('ul', children=[
            [Tag('li', children=['foo']), 
             Tag('li', children=['bar'])]])]), 
     Tag('li', children=['group 2', 
        Tag('ul', children=[
             [Tag('li', children=['baz']), 
              Tag('li', children=['qux']), 
              Tag('li', children=['asdf'])]])])]])

This is a nice way to template HTML, since the resulting code is a lot like an S-expression syntax for the resulting HTML. I still get to use the familiar Python list comprehensions and I save myself the need to pull in another templating library when Twisted already has one.

Klein has some kind of widget architecture, but it never actually appears in the documentation (ever). You wouldn't know it existed unless you read the source code. I wasn't able to work out how or why these widgets existed, because they don't solve the problem above, as far as I could tell. Instead, what I discovered was that the plating system relies on "slots" defined in the templates for where results are embedded. In the above case for the generated HTML this works. The real problem is in trying to view the JSON for such a route results in exception about the inability to serialize a Tag to JSON (which makes some sense, it is a bare Python object).

With the confluence of these two problems you may see the annoyance of dealing with Klein:

An Escape Hatch

It took a bit of head scratching before I dug more deeply into the code and came up with the following solution.

The JSON views will encode the returned dictionary from any function decorated with the routed function — however, prior to that, the _asJSON helper method removes any keys designated as presentationSlots. So to avoid the problem demonstrated above, I just marked the HTML slot as a presentationSlot and returned the data dictionary unaltered.

Won't Work

myStyle = Plating(
    tags=tags.html(
        tags.div(slot(Plating.CONTENT))))

@myStyle.routed(
    app.route("/"),
    tags.div(slot("formatted")))
def root(request):
    return {'formatted': grouper({
        'group 1': ['foo', 'bar'],
        'group 2': ['baz', 'qux', 'asdf']
    })}

Will Work

By adding a keyword argument to the Plating class, presentation_slots={'formatted'}, any slot named "formatted" will be ignored from the JSON serialization process. So leaving the templating function unchanged results in:

myStyle = Plating(
    tags=tags.html(
        tags.div(slot(Plating.CONTENT))),
    presentation_slots={'formatted'})

@myStyle.routed(
    app.route("/"),
    tags.div(slot("formatted")))
def root(request):
    groups = {
        'group 1': ['foo', 'bar'],
        'group 2': ['baz', 'qux', 'asdf']
    }
    return {'formatted': grouper(groups),
            'groups': groups}

It is necessary to add a new keyword argument to the return value, otherwise the JSON view will be empty (having removed formatted entirely). Appending ?json=true to the URL results in the following JSON:

{"groups": {"group 1": ["foo", "bar"], "group 2": ["baz", "qux", "asdf"]}}

I can't claim it is the "right way" to do this, but I can say it worked for my case.

Last Minute Changes

Another place where Twisted really shines is the clean separation of parts. As a good example of this, after having written most of the application assuming HTTP and AMQP my boss messaged me to say "Your thing uses HTTPS, right?". While this might have been troublesome with another library or framework, with Twisted I replaced a call to a TCPServerEndpint with a SSL4ServerEndpint and a "context" pointing to a certificate and key on disk and messaged back "yes". Because Twisted relies so heavily on defined interfaces it isn't a problem to mix and match pieces like legos.

Contributing to Twisted

In using Twisted's asynchronous database API I noticed one small annoyance: when using the "noisy" log-level during the connection to the database, the connection arguments are logged — meaning user name and password information. Twisted is to my mind, one of the better organized open-source projects for new contributors. They clearly outline how to write a patch and the best way to have it accepted. I went ahead and opened a ticket, cloned the repository and committed a fix. The extensive unit test suite passed in both my local repository and after my pull request was submitted. The level of automation and care that is taken really makes the developer experience shine. While I'm still waiting to have it merged, I would happily contribute more fixes in the future based solely on this experience.

[nolan@nprescott.com] $> █