I previously worked out some safer defaults for services I run on a Debian server exposed to the internet. Here are some further notes on really going overboard.
I previously said:
At some point I should probably get around to doing the full container bit and try out systemd-nspawn. I think there is more work involved in setting up the filesystem than I am quite ready for, to say nothing of the networking configuration involved.
To tell the truth, I haven't bothered figuring out nspawn because I
found something I like better. It all began with the discovery of a
tool to audit the security of systemd
services, systemd-analyze
. It is a rudimentary tool
that calls out every security adjacent setting that you might
configure that could have an effect on the service. It
provides a score that does not reflect a real assessment of the
particular service and is instead best viewed as a reminder of those
things you should consider.
I took things to the extreme and tried enabling every single security option.
$ cat /etc/systemd/system/a-server.service
[Unit]
Description=an internet-facing server
[Service]
RootDirectory=/srv/jail
User=my-server-user
Restart=always
Type=simple
CapabilityBoundingSet=
RestrictAddressFamilies=AF_INET
ProtectControlGroups=yes
PrivateTmp=yes
PrivateDevices=yes
PrivateUsers=yes
ProtectControlGroups=yes
ProtectHome=yes
ProtectHostname=yes
ProtectClock=yes
ProtectKernelLogs=yes
ProtectKernelModules=yes
ProtectKernelTunables=yes
ProtectProc=invisible
ProtectSystem=strict
ProcSubset=pid
RestrictNamespaces=yes
RestrictRealtime=yes
NoNewPrivileges=yes
MemoryDenyWriteExecute=yes
SystemCallArchitectures=native
LockPersonality=yes
RestrictSUIDSGID=yes
RemoveIPC=yes
UMask=177
SystemCallFilter=~@clock @debug @module @reboot @privileged @cpu-emulation @obsolete @mount @resources
ReadWritePaths=/srv/jail/var/log/server/access.log /srv/jail/var/log/server/error.log
ExecStart=/opt/server --config /etc/server/config.conf
[Install]
WantedBy=multi-user.target
By just running systemd-analyze security <service
name>
I solicited a report of all the potential knobs to
tweak (which is to say I didn't come up with that big long list all
by myself). Armed
with the
manual I set to reading about each suggestion or highlighted
issue. My understanding of the net effect of the above configuration
is as follows:
chroot
Personally I think the last two are the most significant in
improving the attack surface of a program. Marking the file system
as read only is accomplished via
the ProtectSystem=strict
directive, because the program
requires write access to the two log files (access.log
and
error.log
) there is an exemption made
via ReadWritePaths
.
Interestingly, the call to ExecStart
is relative to the
root directory, but the paths laid out in
the ReadWritePaths
directive are absolute to the host
system.
To be totally honest, I put off looking into nspawn and further
containerization technologies because I tend to find them very
onerous to develop against and tedious to maintain. It was a minor
revelation to find systemd supports more classic isolation via
chroots, identified with the RootDirectory
directive.
The goal immediately became trying things out in a chroot to see how
similar the result was to flying by the seat of my pants and
launching services willy-nilly. To create the chroot I opted to
use debootstrap
because I am on Debian. The entire
process looks like this:
# debootstrap --variant=minbase stable /srv/jail
And that is it. There's nothing much in the chroot yet but it is entirely possible to launch a shell and poke around with that done. In this case I am interested in running a statically-linked go program with a single configuration file to support it -- there is no fancy provisioning necessary, I just copied the file into the new root like so:
# cp /opt/server /srv/jail/opt/server
Once the same is done for the configuration file the whole thing just works.
As nice as a simple little go program is for the above case I started to wonder if this was too good to be true. Loads of little programs start out in one or more scripting languages and those are probably the ones I should be especially concerned about isolating. What is the story like for "deploying" something like, say, a TCL SMTP server inside a chroot like this?
It's all just Debian! I ran apt install tcl tcllib
inside the chroot and all of my programs work as normal from there
on. Of course, if the program does much of anything interesting like
write to a SQLite database the writable file or directory has to be
added to the configuration.
I'm not really sure if it is a great idea to run multiple programs inside a "jail" like this, the chroot for Debian is on the larger side at around ~250Mb so you wouldn't exactly create a million of them. It is possible to use something like Alpine Linux instead, which begins around 6Mb instead but the lack of glibc appeared aggravating enough that I'm happy to live with the larger chroot for now.
I am generally pleased with the outcome here. I think my server is more secure and I feel like I have a thorough understanding of how the various security features piece together (certainly not the case with many bigger containerization tools). I think the approach is probably a little more old-school but not in a bad way! I am used to dealing with apt (and increasingly systemd), I am happy to leverage that knowledge to maintain a more secure environment.
Where I might have previously felt a nagging sense of "knowing better" about how I deployed things to the internet, I don't really feel that with this setup. Sure, a sufficiently determined hacker can probably still break in but the level of effort has been significantly increased with these changes.