indexpost archiveatom feed syndication feed icon

A Few systemd Niceties


In what I swear is probably the last post about systemd for a while I review a few remaining facets of systemd service files and the broader ecosystem available to administrators.

systemd Specifics

Bind Mounts

I previously found the ReadWritePaths setting available to systemd services, and while it worked I wasn't really clear what was happening behind the scenes. Especially annoying was copying files into my chroot for unclear reasons (well, reasons beyond it was required to make it run). Having spent the better part of the last week dealing with unrelated system administration issues I have a much clearer picture of how to do things.

Rather than copying executables or data into a chroot it is possible to bind mount data from the host onto the nested filesystem. This prevents duplication and allows for pretty fine-grained access control as well. The setting BindReadOnlyPaths allows for executables to be accessed as normal but provides a more restricted view of the system.

I believe the following is analogous where the systemd method is obviously less error-prone.

# unshare -m
unshare the mount namespace to effectively create a new namespace which can be modified freely. With the new namespace bind-mount a read-only version of my program into the chroot:
# mount -o bind,ro /opt/my-program /srv/chroot/opt/my-program

At this point, executing /opt/my-program inside the chroot works as expected but even inside the chroot the root user is unable to modify the executable because it has been mounted read-only. Exiting (first the chroot, then the namespace created by unshare) leaves the system as it was. No need to copy the executable into the chroot or fiddle with permissions.

Logs Directory

Rather than carve out a ReadWritePath within the chroot it turns out systemd supports a LogsDirectory option to ease this very process. In practice it means my logs end up in plain old /var/log/... like everything else and things like log rotation continue to work without extra effort to account for a chroot. There are similar directives for configuration and state but I haven't yet made use of those.

Instead of this:

ReadWritePaths=/srv/chroot/var/log/my-server/access.log /srv/chroot/var/log/my-server/error.log
It is possible to write:

Declarative Users

If the DynamicUser setting isn't quite workable to lock down a service it is advisable to at least use dedicated service accounts. I did this previously the old-fashioned way, creating users and groups with nologin shells and no home directory. It turns out there is an alternative available with sysusers.d. sysusers.d allows for declarative users via configuration file and the systemd-sysusers command.

For a basic service user with no shell or home directory I created a file like this:

$ cat /etc/sysusers.d/foo.conf
#Type Name          ID   GECOS                  Home directory  Shell
u     foo-service   -    "foo service user"

$ sudo systemd-sysusers /etc/sysusers.d/foo.conf

In the case that the user (and corresponding group) does not exist, it is created. The ID will be selected automatically. There are further options available for creating groups and ranges - I haven't had a need for it because of how simple my administration approach tends to be.

More than anything it seems like the declarative, configuration-based approach to administering systemd brings my own administration pretty far towards standardized deployment. It isn't quite to the point where something like Ansible would get you but it does seem remarkably close. Define what a service is, limit the capabilities it has in the broader system, define users, permissions, start-up and restart behavior. All that configuration can be committed to source control and shared or packaged with an application.


I finally got around to reading more about systemd-nspawn, the container integration available with systemd. While it isn't quite so bad as I had imagined I also don't know that I have a particularly good use for it yet. Due to the number of security options I have configured I think my own services may achieve a higher baseline of isolation than simply launching a container could. That isn't to say that systemd-nspawn can't apply those same settings (it can) but rather that I am not doing anything that yet requires a container.

It does seem appealing for easily running alternate distros but I am not entirely sure if non-systemd distros (such as Alpine) can achieve the kind of integration available to Debian, Fedora, etc. As a basic test I tried launching an Alpine Linux container like this and it was dead simple to do some basic tests:

$ wget
$ mkdir /tmp/alpine
$ tar xzf alpine-minirootfs-3.15.0-x86_64.tar.gz -C /tmp/alpine
$ sudo systemd-nspawn --private-users=pick --directory /tmp/alpine

Spawning container alpine on /tmp/alpine.
Press ^] three times within 1s to kill container.
Selected user namespace base 748290048 and range 65536.

alpine:~# cat /etc/os-release
NAME="Alpine Linux"
PRETTY_NAME="Alpine Linux v3.15"

I can see using this for testing or developing with distro-specific packages like Sourcehut (which is primarily distributed as Alpine packages). I don't yet entirely understand the implications for some of the user-mapping options and am especially wary of using root inside of containers.

I had previously thought that networking would be a real pain for containers. Instead the default is for the container to use the host networking unless otherwise configured. It is of course a good idea to carve up a private network and segment what a container can access and how. This is where all of the complexity is though and I haven't bothered with it yet.

Thinking More Broadly

Packaging Done Smarter

I've been experimenting more with TCL and find the tclkit facilities very neat. Built on top of the virtual filesystem capabilities of TCL it is possible to construct a single executable capable of running arbitrary TCL code on machines without having TCL installed. Additionally you can "wrap" tclkits to produce your own executables with interpreter, libraries, application code, and assets.

This addresses a major pain point I have had with Python in a way that is surprisingly elegant. While there are a few Python projects to "freeze" or "wrap" programs in a similar way I have not had very good experiences with them. Either paths get confused, permissions are screwy, the executables are enormous — things just are not painless enough to bother with. Sure, it is possible to run most Python projects inside a virtual environment but in practice this always seems to make the deploy targets more like development environments than anything. It seems the only distribution mechanism for Python programs that anyone seems to consider any more is Docker containers.

I haven't entirely made up my mind about static vs. dynamic linking. There has been much written in favor of each and I don't have the sorts of environments or problems that others do. I can see the appeal of static linking as a midway point between shipping an entire file system container compared to keeping a server up to date with a variety of shared libraries that must accommodate all of the deployed services. That being said, if you try to use a chroot like I have described it recreates several issues present with static linking because the host system updates do no reach the chroot (without some effort, as far as I know).

All I have really come to realize is how much I still prefer my old way of doing things. Lean on operating system and package maintainers for sane defaults and security updates, don't go crazy with deployment pipelines or byzantine artifacts. Giving a modicum of thought to delineating services, users and resources and then codifying those constraints into a service file is probably good enough. Sure, containers or chroots might add another layer of indirection but I am unsure it meaningfully improves the attack surface.

I think my efforts will be better spent doing some thinking up-front and working out how to do things like reduce third-party dependencies from my programs. Even just auditing third-party packages is probably a better use of my time than constructing increasingly inscrutable deployment targets.

My Current Order of Preference

  1. use whatever the operating system provides
  2. use packages provided by your distribution (Debian stable in my case)
  3. use static binaries or pre-packaged builds (e.g. tclkit)
  4. ship a whole file system/container
  5. ship some weird amalgamation of you development environment to bundle your dependencies and hope it works at the deploy target (e.g. virtual environments)


I have done a lot of reading about systemd lately. I find myself largely content with the level of both sophistication and simplicity in my current setup. My only real remaining questions involve packaging and shipping software and configurations for internal services. I have previous experience using Debian packages in a commercial context and it wasn't really that bad. I'm not sure it makes sense to create packages to bundle applications or configuration purely for internal use but it does have a certain appeal to me.

A problem for another day perhaps.