In what I swear is probably the last post about systemd for a while I review a few remaining facets of systemd service files and the broader ecosystem available to administrators.
I previously found the ReadWritePaths setting available
to systemd services, and while it worked I wasn't really clear what
was happening behind the scenes. Especially annoying was copying
files into my chroot for unclear reasons (well, reasons beyond it
was required to make it run). Having spent the better part of the
last week dealing with unrelated system administration issues I have
a much clearer picture of how to do things.
Rather than copying executables or data into a chroot it is possible
to bind mount data from the host onto the nested filesystem. This
prevents duplication and allows for pretty fine-grained access
control as well. The setting BindReadOnlyPaths allows
for executables to be accessed as normal but provides a more
restricted view of the system.
I believe the following is analogous where the systemd method is obviously less error-prone.
# unshare -munshare the mount namespace to effectively create a new
namespace which can be modified freely. With the new namespace
bind-mount a read-only version of my program into the chroot:
# mount -o bind,ro /opt/my-program /srv/chroot/opt/my-program
At this point, executing /opt/my-program inside the
chroot works as expected but even inside the chroot the root user is
unable to modify the executable because it has been mounted
read-only. Exiting (first the chroot, then the namespace created by
unshare) leaves the system as it was. No need to copy the executable
into the chroot or fiddle with permissions.
Rather than carve out a ReadWritePath within the chroot
it turns out systemd supports
a LogsDirectory
option to ease this very process. In practice it means my logs end
up in plain old /var/log/... like everything else and
things like log rotation continue to work without extra effort to
account for a chroot. There are similar directives for configuration
and state but I haven't yet made use of those.
Instead of this:
ReadWritePaths=/srv/chroot/var/log/my-server/access.log /srv/chroot/var/log/my-server/error.logLogsDirectory=my-server
If the DynamicUser setting isn't quite workable to lock
down a service it is advisable to at least use dedicated service
accounts. I did this previously the old-fashioned way, creating
users and groups with nologin shells and no home
directory. It turns out there is an alternative available
with sysusers.d. sysusers.d allows for
declarative users via configuration file and
the systemd-sysusers command.
For a basic service user with no shell or home directory I created a file like this:
$ cat /etc/sysusers.d/foo.conf
#Type Name ID GECOS Home directory Shell
u foo-service - "foo service user"
$ sudo systemd-sysusers /etc/sysusers.d/foo.conf
In the case that the user (and corresponding group) does not exist, it is created. The ID will be selected automatically. There are further options available for creating groups and ranges - I haven't had a need for it because of how simple my administration approach tends to be.
More than anything it seems like the declarative, configuration-based approach to administering systemd brings my own administration pretty far towards standardized deployment. It isn't quite to the point where something like Ansible would get you but it does seem remarkably close. Define what a service is, limit the capabilities it has in the broader system, define users, permissions, start-up and restart behavior. All that configuration can be committed to source control and shared or packaged with an application.
I finally got around to reading more about systemd-nspawn, the container integration available with systemd. While it isn't quite so bad as I had imagined I also don't know that I have a particularly good use for it yet. Due to the number of security options I have configured I think my own services may achieve a higher baseline of isolation than simply launching a container could. That isn't to say that systemd-nspawn can't apply those same settings (it can) but rather that I am not doing anything that yet requires a container.
It does seem appealing for easily running alternate distros but I am not entirely sure if non-systemd distros (such as Alpine) can achieve the kind of integration available to Debian, Fedora, etc. As a basic test I tried launching an Alpine Linux container like this and it was dead simple to do some basic tests:
$ wget https://dl-cdn.alpinelinux.org/alpine/v3.15/releases/x86_64/alpine-minirootfs-3.15.0-x86_64.tar.gz
$ mkdir /tmp/alpine
$ tar xzf alpine-minirootfs-3.15.0-x86_64.tar.gz -C /tmp/alpine
$ sudo systemd-nspawn --private-users=pick --directory /tmp/alpine
Spawning container alpine on /tmp/alpine.
Press ^] three times within 1s to kill container.
Selected user namespace base 748290048 and range 65536.
alpine:~# cat /etc/os-release
NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.15.0
PRETTY_NAME="Alpine Linux v3.15"
HOME_URL="https://alpinelinux.org/"
BUG_REPORT_URL="https://bugs.alpinelinux.org/"
I can see using this for testing or developing with distro-specific packages like Sourcehut (which is primarily distributed as Alpine packages). I don't yet entirely understand the implications for some of the user-mapping options and am especially wary of using root inside of containers.
I had previously thought that networking would be a real pain for containers. Instead the default is for the container to use the host networking unless otherwise configured. It is of course a good idea to carve up a private network and segment what a container can access and how. This is where all of the complexity is though and I haven't bothered with it yet.
I've been experimenting more with TCL and find the tclkit facilities very neat. Built on top of the virtual filesystem capabilities of TCL it is possible to construct a single executable capable of running arbitrary TCL code on machines without having TCL installed. Additionally you can "wrap" tclkits to produce your own executables with interpreter, libraries, application code, and assets.
This addresses a major pain point I have had with Python in a way that is surprisingly elegant. While there are a few Python projects to "freeze" or "wrap" programs in a similar way I have not had very good experiences with them. Either paths get confused, permissions are screwy, the executables are enormous — things just are not painless enough to bother with. Sure, it is possible to run most Python projects inside a virtual environment but in practice this always seems to make the deploy targets more like development environments than anything. It seems the only distribution mechanism for Python programs that anyone seems to consider any more is Docker containers.
I haven't entirely made up my mind about static vs. dynamic
linking. There has
been much written in
favor of each and I don't have the sorts of environments or problems
that others do. I can see the appeal of static linking as a midway
point between shipping an entire file system
container compared to keeping a server up to date with a variety of
shared libraries that must accommodate all of the deployed
services. That being said, if you try to use a chroot like I have
described it recreates several issues present with static linking
because the host system updates do no reach the chroot (without some
effort, as far as I know).
All I have really come to realize is how much I still prefer my old way of doing things. Lean on operating system and package maintainers for sane defaults and security updates, don't go crazy with deployment pipelines or byzantine artifacts. Giving a modicum of thought to delineating services, users and resources and then codifying those constraints into a service file is probably good enough. Sure, containers or chroots might add another layer of indirection but I am unsure it meaningfully improves the attack surface.
I think my efforts will be better spent doing some thinking up-front and working out how to do things like reduce third-party dependencies from my programs. Even just auditing third-party packages is probably a better use of my time than constructing increasingly inscrutable deployment targets.
I have done a lot of reading about systemd lately. I find myself largely content with the level of both sophistication and simplicity in my current setup. My only real remaining questions involve packaging and shipping software and configurations for internal services. I have previous experience using Debian packages in a commercial context and it wasn't really that bad. I'm not sure it makes sense to create packages to bundle applications or configuration purely for internal use but it does have a certain appeal to me.
A problem for another day perhaps.