Just a few notes from the trenches in keeping a public-facing web server up and running. The quality of freely available software is pretty amazing.
A few months ago I received a notice from my hosting provider:
We have detected a system issue with the node hosting the instances listed above. Our engineering team applied system updates and scheduled a brief maintenance window to perform a node restart.
I tend to defer to automated processes for most of my server upkeep, so the above got me thinking about exactly how long this server has been happily humming along:
It turns out, I hadn't performed a restart since the last time I upgraded the operating system, back in 2017. Through the use of unattended-upgrades and Debian's security repositories I was already running recent releases of all of the software on the server, it was only the kernel that hadn't applied changes in that time.
I am, of course, familiar with spectre, meltdown etc. ad nauseam, but due to the nature of the attacks (as I understand it), restarting would not prevent the vulnerability because of the way web-hosting works. Along with the fact that I don't run other people's code (directly), I figured the hosting provider would eventually identify the appropriate mitigation at the level of the hypervisor.
I've written before about a few different pieces of automation that still exist in one form or another to help with keeping things running around here. Coupled with certbot for automated HTTPS certificate renewals, the server pretty much runs itself.
Reading about the release of Debian 10.2
today I figured I'd take
some time to perform an operating system upgrade. I've been happily
running Debian 9 (Stretch) since migrating off of Digital Ocean. Since
I'm still happy with the combination of price and performance on Vultr
I was interested in keeping the machine largely as-is. It has been a
while since I've performed an operating system upgrade that wasn't
problematic (e.g. any Mac OS upgrade). Consequently I was pleasantly
surprised at how painless the dist-upgrade process was. I made a few
changes to the code-name in my /etc/apt/sources.list file as
outlined in the
instructions
and walked through a few minor changes to configuration files.
And that was it.
After verifying everything was running smoothly (without so much as downtime on this website!), I went ahead and rebooted to include the latest kernel updates for Debian 10.
I've explored OpenBSD before as a potential alternative for this server and I think the system as a whole is more comprehensible to a part-time system administrator like myself when compared to the Linux ecosystem. That being said, the quality of Debian and the care taken with their documentation will probably keep me around at least until the next time I have to switch machines.