Further Isolation, systemd2022-02-22
I previously worked out some safer defaults for services I run on a Debian server exposed to the internet. Here are some further notes on really going overboard.
I previously said:
At some point I should probably get around to doing the full container bit and try out systemd-nspawn. I think there is more work involved in setting up the filesystem than I am quite ready for, to say nothing of the networking configuration involved.
To tell the truth, I haven't bothered figuring out nspawn because I
found something I like better. It all began with the discovery of a
tool to audit the security of systemd
systemd-analyze. It is a rudimentary tool
that calls out every security adjacent setting that you might
configure that could have an effect on the service. It
provides a score that does not reflect a real assessment of the
particular service and is instead best viewed as a reminder of those
things you should consider.
I took things to the extreme and tried enabling every single security option.
$ cat /etc/systemd/system/a-server.service [Unit] Description=an internet-facing server [Service] RootDirectory=/srv/jail User=my-server-user Restart=always Type=simple CapabilityBoundingSet= RestrictAddressFamilies=AF_INET ProtectControlGroups=yes PrivateTmp=yes PrivateDevices=yes PrivateUsers=yes ProtectControlGroups=yes ProtectHome=yes ProtectHostname=yes ProtectClock=yes ProtectKernelLogs=yes ProtectKernelModules=yes ProtectKernelTunables=yes ProtectProc=invisible ProtectSystem=strict ProcSubset=pid RestrictNamespaces=yes RestrictRealtime=yes NoNewPrivileges=yes MemoryDenyWriteExecute=yes SystemCallArchitectures=native LockPersonality=yes RestrictSUIDSGID=yes RemoveIPC=yes UMask=177 SystemCallFilter=~@clock @debug @module @reboot @privileged @cpu-emulation @obsolete @mount @resources ReadWritePaths=/srv/jail/var/log/server/access.log /srv/jail/var/log/server/error.log ExecStart=/opt/server --config /etc/server/config.conf [Install] WantedBy=multi-user.target
By just running
systemd-analyze security <service
name> I solicited a report of all the potential knobs to
tweak (which is to say I didn't come up with that big long list all
by myself). Armed
manual I set to reading about each suggestion or highlighted
issue. My understanding of the net effect of the above configuration
is as follows:
- runs as a dedicated (non-privileged) user
- no access to control groups
- isolated temporary directory
- isolated device directory
- no access to change system host name
- no access to adjust system clock
- no ability to load kernel modules
- no visibility to the /proc file system
- no access to control namespaces
- no access to change privileges
- no ability to execute writable memory
- no visibility to other users on the system
- no visibility to home directories
- entire file system (save the two log files) is read-only
- entire program runs in a
Personally I think the last two are the most significant in
improving the attack surface of a program. Marking the file system
as read only is accomplished via
ProtectSystem=strict directive, because the program
requires write access to the two log files (
error.log) there is an exemption made
Interestingly, the call to
ExecStart is relative to the
root directory, but the paths laid out in
ReadWritePaths directive are absolute to the host
To be totally honest, I put off looking into nspawn and further
containerization technologies because I tend to find them very
onerous to develop against and tedious to maintain. It was a minor
revelation to find systemd supports more classic isolation via
chroots, identified with the
The goal immediately became trying things out in a chroot to see how
similar the result was to flying by the seat of my pants and
launching services willy-nilly. To create the chroot I opted to
debootstrap because I am on Debian. The entire
process looks like this:
# debootstrap --variant=minbase stable /srv/jail
And that is it. There's nothing much in the chroot yet but it is entirely possible to launch a shell and poke around with that done. In this case I am interested in running a statically-linked go program with a single configuration file to support it -- there is no fancy provisioning necessary, I just copied the file into the new root like so:
# cp /opt/server /srv/jail/opt/server
Once the same is done for the configuration file the whole thing just works.
What About Interpreted Programs?
As nice as a simple little go program is for the above case I started to wonder if this was too good to be true. Loads of little programs start out in one or more scripting languages and those are probably the ones I should be especially concerned about isolating. What is the story like for "deploying" something like, say, a TCL SMTP server inside a chroot like this?
It's all just Debian! I ran
apt install tcl tcllib
inside the chroot and all of my programs work as normal from there
on. Of course, if the program does much of anything interesting like
write to a SQLite database the writable file or directory has to be
added to the configuration.
I'm not really sure if it is a great idea to run multiple programs inside a "jail" like this, the chroot for Debian is on the larger side at around ~250Mb so you wouldn't exactly create a million of them. It is possible to use something like Alpine Linux instead, which begins around 6Mb instead but the lack of glibc appeared aggravating enough that I'm happy to live with the larger chroot for now.
I am generally pleased with the outcome here. I think my server is more secure and I feel like I have a thorough understanding of how the various security features piece together (certainly not the case with many bigger containerization tools). I think the approach is probably a little more old-school but not in a bad way! I am used to dealing with apt (and increasingly systemd), I am happy to leverage that knowledge to maintain a more secure environment.
Where I might have previously felt a nagging sense of "knowing better" about how I deployed things to the internet, I don't really feel that with this setup. Sure, a sufficiently determined hacker can probably still break in but the level of effort has been significantly increased with these changes.