In doing some greenfield development at work recently I've been digging into Twisted and the Klein "micro-framework". I'm really coming to appreciate some aspects of the projects as a result.
The project I'm working on is a stand-alone utility to, primarily, surface information via a series of web-views in both a human-readable and machine-readable format and secondarily, to subscribe to a set of updates via message queue and write them to a database.
I feel like this is an under-appreciated aspect of Twisted, I know I
only discovered it in earnest when I first made my system monitoring
service, but the application
and service
architecture
makes it very easy to combine discrete components in a easily
separable way. In this instance I wanted to an HTTP server to surface
both kinds of web-views, and an AMQP client to interface with the
message queue. In my particular case, the both the HTTP server and the
AMQP client relied on access to the same database. It was a simple
thing to establish an asynchronous connection pool with Twisted
adbapi abstracting both interfaces behind a single class. It was a
cinch to get all three pieces, shared database access, client, and
server running under the same process.
As part of the dual-format presentation (human-readable and machine-readable), I wanted to minimize the duplication without having to go "full single page application". Part of what led me to choose Klein was the Plating module, which ties in a basic templating system along with an automatic JSON interface for each route. I'll be the first to admit that Klein isn't widely used, there aren't many answers to be found on StackOverflow and no one at work had used it before. This would be a pretty big downside but for the fact that it is very easy to just read the source and answer questions on my own.
As a particular example, the plating mechanism is pretty rudimentary in handling nested data. I had a data-set that looked something like the following:
'groups': {
'group 1': ['foo', 'bar'],
'group 2': ['baz', 'qux', 'asdf']
...
}
And wanted to surface it into an HTML view like the following:
The automatic handling of iterable data first "flattens" things, so
there wasn't a good way to display the above with just the
render='groups:list' format that the documentation describes. It is
a pretty simple matter to instead design a function to handle the data
myself:
def grouper(data_dictionary):
return tags.ul(
[tags.li(group)(
tags.ul(
[tags.li(item) for item in data_dictionary[group]]))
for group in data_dictionary])
The above is results in a data structure like this (with some whitespace added for clarity):
Tag('ul', children=[
[Tag('li', children=['group 1',
Tag('ul', children=[
[Tag('li', children=['foo']),
Tag('li', children=['bar'])]])]),
Tag('li', children=['group 2',
Tag('ul', children=[
[Tag('li', children=['baz']),
Tag('li', children=['qux']),
Tag('li', children=['asdf'])]])])]])
This is a nice way to template HTML, since the resulting code is a lot like an S-expression syntax for the resulting HTML. I still get to use the familiar Python list comprehensions and I save myself the need to pull in another templating library when Twisted already has one.
Klein has some kind of widget architecture, but it never actually
appears in the documentation (ever). You wouldn't know it existed
unless you read the source
code. I
wasn't able to work out how or why these widgets existed, because they
don't solve the problem above, as far as I could tell. Instead, what I
discovered was that the plating system relies on "slots" defined in
the templates for where results are embedded. In the above case for
the generated HTML this works. The real problem is in trying to view
the JSON for such a route results in exception about the inability to
serialize a Tag to JSON (which makes some sense, it is a bare Python
object).
With the confluence of these two problems you may see the annoyance of dealing with Klein:
It took a bit of head scratching before I dug more deeply into the code and came up with the following solution.
The JSON views will encode the returned dictionary from any function
decorated with the routed function — however, prior to that, the
_asJSON helper method removes any keys designated as
presentationSlots. So to avoid the problem demonstrated above, I
just marked the HTML slot as a presentationSlot and returned the
data dictionary unaltered.
myStyle = Plating(
tags=tags.html(
tags.div(slot(Plating.CONTENT))))
@myStyle.routed(
app.route("/"),
tags.div(slot("formatted")))
def root(request):
return {'formatted': grouper({
'group 1': ['foo', 'bar'],
'group 2': ['baz', 'qux', 'asdf']
})}
By adding a keyword argument to the Plating class,
presentation_slots={'formatted'}, any slot named "formatted" will be
ignored from the JSON serialization process. So leaving the templating
function unchanged results in:
myStyle = Plating(
tags=tags.html(
tags.div(slot(Plating.CONTENT))),
presentation_slots={'formatted'})
@myStyle.routed(
app.route("/"),
tags.div(slot("formatted")))
def root(request):
groups = {
'group 1': ['foo', 'bar'],
'group 2': ['baz', 'qux', 'asdf']
}
return {'formatted': grouper(groups),
'groups': groups}
It is necessary to add a new keyword argument to the return value,
otherwise the JSON view will be empty (having removed formatted
entirely). Appending ?json=true to the URL results in the following
JSON:
{"groups": {"group 1": ["foo", "bar"], "group 2": ["baz", "qux", "asdf"]}}
I can't claim it is the "right way" to do this, but I can say it worked for my case.
Another place where Twisted really shines is the clean separation of
parts. As a good example of this, after having written most of the
application assuming HTTP and AMQP my boss messaged me to say "Your
thing uses HTTPS, right?". While this might have been troublesome with
another library or framework, with Twisted I replaced a call to a
TCPServerEndpint with a SSL4ServerEndpint and a "context" pointing
to a certificate and key on disk and messaged back "yes". Because
Twisted relies so heavily on defined interfaces it isn't a problem to
mix and match pieces like legos.
In using Twisted's asynchronous database API I noticed one small annoyance: when using the "noisy" log-level during the connection to the database, the connection arguments are logged — meaning user name and password information. Twisted is to my mind, one of the better organized open-source projects for new contributors. They clearly outline how to write a patch and the best way to have it accepted. I went ahead and opened a ticket, cloned the repository and committed a fix. The extensive unit test suite passed in both my local repository and after my pull request was submitted. The level of automation and care that is taken really makes the developer experience shine. While I'm still waiting to have it merged, I would happily contribute more fixes in the future based solely on this experience.