Further Isolation, systemd

I previously worked out some safer defaults for services I run on a Debian server exposed to the internet. Here are some further notes on really going overboard.

I previously said:

At some point I should probably get around to doing the full container bit and try out systemd-nspawn. I think there is more work involved in setting up the filesystem than I am quite ready for, to say nothing of the networking configuration involved.

To tell the truth, I haven't bothered figuring out nspawn because I found something I like better. It all began with the discovery of a tool to audit the security of systemd services, systemd-analyze. It is a rudimentary tool that calls out every security adjacent setting that you might configure that could have an effect on the service. It provides a score that does not reflect a real assessment of the particular service and is instead best viewed as a reminder of those things you should consider.

I took things to the extreme and tried enabling every single security option.

$ cat /etc/systemd/system/a-server.service
[Unit]
Description=an internet-facing server

[Service]
RootDirectory=/srv/jail
User=my-server-user
Restart=always
Type=simple
CapabilityBoundingSet=
RestrictAddressFamilies=AF_INET
ProtectControlGroups=yes
PrivateTmp=yes
PrivateDevices=yes
PrivateUsers=yes
ProtectControlGroups=yes
ProtectHome=yes
ProtectHostname=yes
ProtectClock=yes
ProtectKernelLogs=yes
ProtectKernelModules=yes
ProtectKernelTunables=yes
ProtectProc=invisible
ProtectSystem=strict
ProcSubset=pid
RestrictNamespaces=yes
RestrictRealtime=yes
NoNewPrivileges=yes
MemoryDenyWriteExecute=yes
SystemCallArchitectures=native
LockPersonality=yes
RestrictSUIDSGID=yes
RemoveIPC=yes
UMask=177
SystemCallFilter=~@clock @debug @module @reboot @privileged @cpu-emulation @obsolete @mount @resources
ReadWritePaths=/srv/jail/var/log/server/access.log /srv/jail/var/log/server/error.log 
ExecStart=/opt/server --config /etc/server/config.conf

[Install]
WantedBy=multi-user.target

By just running systemd-analyze security <service name> I solicited a report of all the potential knobs to tweak (which is to say I didn't come up with that big long list all by myself). Armed with the manual I set to reading about each suggestion or highlighted issue. My understanding of the net effect of the above configuration is as follows:

• runs as a dedicated (non-privileged) user
• no access to control groups
• isolated temporary directory
• isolated device directory
• no access to change system host name
• no access to adjust system clock
• no ability to load kernel modules
• no visibility to the /proc file system
• no access to control namespaces
• no access to change privileges
• no ability to execute writable memory
• no visibility to other users on the system
• no visibility to home directories
• entire file system (save the two log files) is read-only
• entire program runs in a chroot

Personally I think the last two are the most significant in improving the attack surface of a program. Marking the file system as read only is accomplished via the ProtectSystem=strict directive, because the program requires write access to the two log files (access.log and error.log) there is an exemption made via ReadWritePaths.

Interestingly, the call to ExecStart is relative to the root directory, but the paths laid out in the ReadWritePaths directive are absolute to the host system.

chroot

To be totally honest, I put off looking into nspawn and further containerization technologies because I tend to find them very onerous to develop against and tedious to maintain. It was a minor revelation to find systemd supports more classic isolation via chroots, identified with the RootDirectory directive.

The goal immediately became trying things out in a chroot to see how similar the result was to flying by the seat of my pants and launching services willy-nilly. To create the chroot I opted to use debootstrap because I am on Debian. The entire process looks like this:

# debootstrap --variant=minbase stable /srv/jail

And that is it. There's nothing much in the chroot yet but it is entirely possible to launch a shell and poke around with that done. In this case I am interested in running a statically-linked go program with a single configuration file to support it -- there is no fancy provisioning necessary, I just copied the file into the new root like so:

# cp /opt/server /srv/jail/opt/server

Once the same is done for the configuration file the whole thing just works.

What About Interpreted Programs?

As nice as a simple little go program is for the above case I started to wonder if this was too good to be true. Loads of little programs start out in one or more scripting languages and those are probably the ones I should be especially concerned about isolating. What is the story like for "deploying" something like, say, a TCL SMTP server inside a chroot like this?

It's all just Debian! I ran apt install tcl tcllib inside the chroot and all of my programs work as normal from there on. Of course, if the program does much of anything interesting like write to a SQLite database the writable file or directory has to be added to the configuration.

I'm not really sure if it is a great idea to run multiple programs inside a "jail" like this, the chroot for Debian is on the larger side at around ~250Mb so you wouldn't exactly create a million of them. It is possible to use something like Alpine Linux instead, which begins around 6Mb instead but the lack of glibc appeared aggravating enough that I'm happy to live with the larger chroot for now.

Thoughts

I am generally pleased with the outcome here. I think my server is more secure and I feel like I have a thorough understanding of how the various security features piece together (certainly not the case with many bigger containerization tools). I think the approach is probably a little more old-school but not in a bad way! I am used to dealing with apt (and increasingly systemd), I am happy to leverage that knowledge to maintain a more secure environment.

Where I might have previously felt a nagging sense of "knowing better" about how I deployed things to the internet, I don't really feel that with this setup. Sure, a sufficiently determined hacker can probably still break in but the level of effort has been significantly increased with these changes.