indexpost archiveatom feed syndication feed icon

Zero Downtime Upgrades


I upgraded one of my machines to the development version of Fedora Linux and thought it may prove fruitful to try out some of the experimental features. While I haven't entirely grasped how to use sysupdate I did learn a few neat tricks.

sysupdate is a new feature (as of May at least) that is considered experimental. The intent seems to be partially automated updates via systemd timers, it can pull from remote HTTP servers, local directories etc. It seems to be a pretty obvious pull-based mechanism that allows for maintaining multiple versions on the same machine (but only allows running a single version at a time). It looks like the intent is to "pull" some software package/image/container as package_1 and later when an update is available sysupdate will pull package_2 and you would maintain both copies locally (the exact number is configurable) with the name package being symlinked to the newest version, based on the trailing numeric value.

For me personally I was interested in the overlap between this and portable services which I took a look at for a dumb April Fool's Day joke. My idea (and I think the intent in their design) is to package all sorts of typically standalone software in something container-like (and sometimes an actual container). The integration with systemd means things like service files, sockets and timers can all be included and when the portable service is "attached" they are all automatically plugged in. My limited experience has been that things just work and it is pretty easy to get a reasonably secure plug-and-play setup for a few different server-type softwares.

There is a minor annoyance with the portable services setup though that I was a little flummoxed by which is the manner in which socket files are unloaded and thus defeat some of the benefits of socket activation. I mostly understand why this is the case but it means a socket-activated server packaged with the binary, service file and socket file will suffer some small amount of downtime when things are detached and reattached. There is however this quote in the portable services documentation:

The portablectl reattach command combines a detach with an attach. It is useful in case an image gets upgraded, as it allows performing a restart operation on the units instead of stop plus start, thus providing lower downtime and avoiding losing runtime state associated with the unit such as the file descriptor store.

It at least sounded like a better software upgrade path was available.

One Solution

While it breaks some of the encapsulation built into portable services I found that by moving the socket files associated with my server out of the portable service I was able to guarantee better availability during software upgrades. The server in question is taken nearly directly from this introductory post from Lennart Poettering, it is a simple little HTTP server that uses a local file to count visitors. All it does is accept an HTTP request, open a file and increment a counter before responding with "Hello! You are visitor #%d!". The server isn't the interesting part of this! The necessary service file included with the portable service looks like this:

Description=Portable Walkthrough Go Edition Service


Key to my finding though is that portable-walkthrough-go.socket is not included in the portable service package. I wrote it to the host server directly and it looks like this:

Description=Portable Walkthrough Go Edition Socket



With this one modification it is possible for systemd to continue listening on the designated port and buffer while the backing server is restarted. For my own testing I created a second version of the server that responds with "Hola! You are visitor #%d!" so that it is obvious if and when the new software is working.

Because I am a dinosaur I still use apache bench for this sort of testing. Specifically I set apache bench to run a basic load test for a fixed amount of time, pointed at my systemd socket. While the load test executed I performed the sort of upgrade described above, symlinking my newer software and then running portablectl reattach --now portable-walkthrough-go.

The results were pretty gratifying, mid-test the logs show the following:

LOG: header received:
HTTP/1.0 200 OK
Date: Sat, 03 Sep 2022 23:06:32 GMT
Content-Length: 32
Content-Type: text/plain; charset=utf-8

Hello! You are visitor #120221!

LOG: header received:
HTTP/1.0 200 OK
Date: Sat, 03 Sep 2022 23:06:32 GMT
Content-Length: 31
Content-Type: text/plain; charset=utf-8

Hola! You are visitor #120222!

With no service interruption the new software version rolls out and quietly starts working. The test report reiterates that no failures occurred as a result of the deployment:

Concurrency Level:      10
Time taken for tests:   30.002 seconds
Complete requests:      28988
Failed requests:        0
Total transferred:      4296068 bytes
HTML transferred:       904472 bytes
Requests per second:    966.20 [#/sec] (mean)


As with all things, there are a few gotchas. Firstly, the above test does not make use of TCP keepalive, doing so caused ~16 failed requests. The good news though is this is fixable by sticking it all behind a load balancer like HAProxy. By fronting the requests with the load balancer keepalives will be answered while the server restarts and the socket can buffer requests so none are dropped. This is super cool to me.

In the same vein long-lived connections like a websocket would likely suffer a similar failure without such an easy fix. This isn't a go/haproxy/systemd problem though it's just how long-lived connections tend to work in my experience. The only work around I'm familiar with tends to leave worker processes running so long as connections exist and serve new requests from the upgraded software, which has always been fraught with operational snags for me.

While I am happy with my findings so far I was stumped by how such a workflow of reattaching a portable service was intended to be integrated with sysupdate. The image/package is pulled but then what? Truth be told I haven't figured it out. It seems I'm not the only one with this question though, there is an unanswered email to the systemd-devel mailing list asking as much. It seems there isn't a totally automatic way to roll out updates to portable services but this is rapidly approaching good enough for me — perfect is the enemy of good after all.

I will keep an eye out for automated techniques for triggering restarts for portable services but in the mean time I can take my networking Swiss army knife, a stock Linux machine and get zero-downtime, reasonably secure internet services with a simple deployment story.