A Neat Nginx Configuration

A real world use case for an interesting Nginx configuration presented itself recently. Some notes on proxying multiple non-HTTP sites and servers on one machine.

The Setup

I have been consolidating some different spaces online down to a single machine (this one) and needed to work out two different pieces. The first is pretty standard, a second HTTP site hosted from the existing Nginx server. The second was slightly more interesting and led me into a few configuration options I haven't previously needed — namely, reverse proxying a single port to two different servers under different domain names.

In my particular case I wanted to consolidate two different non-HTTP servers using the same port to a single machine, without modifying the public-facing configuration (i.e. keeping the single port for both domains). This is probably most easily explained with a diagram:

The motivation is probably obvious, it keeps URIs free of explicit port configurations by maintaining the default for any given protocol. Similarly, by consolidating down to a single machine I can save money and better utilize the resources available. In my case I tend to under utilize this machine which is already the smallest available on my hosting provider (512MB of RAM, 1 CPU); it'd be nice to piggyback a few different sites and minimize the number of possible failure points.

Nginx Stream Module

The key piece to proxying non-HTTP connections across domains is the Nginx stream module, which is not enabled by default in Debian's Nginx package. It is available through the nginx-extras package in APT though, so it is no great burden to install. A little confusingly, the Nginx module system requires explicit loading, so despite being installed, the stream module cannot be used until it is loaded inside of /etc/nginx/nginx.conf with the following:

load_module /usr/lib/nginx/modules/ngx_stream_module.so;

With that loaded though, the stream configuration is a relative breeze:

stream {
    upstream foo {
        server 127.0.0.1:1235;
    }

    upstream bar {
        server 127.0.0.1:1236;
    }

    map $ssl_preread_server_name $name {
        foo.com foo;
        bar.com bar;
    }

    server {
        ssl_preread on;
        listen 1234;
        proxy_pass $name;
    }
}

A map is defined relating domain names to upstream configurations based on the hostname. This is really a case of virtual hosting, which explains the use of $ssl_preread_server_name, which uses SNI. I think something similar might be possible with non-TLS connections with less fuss, but the above works and is necessary for my case of a non-HTTP TLS connection.

One annoying thing that caught me out is that the stream "block" may not appear inside of a server or http block inside of the Nginx configuration. In my case that meant it was at the topmost level and not contained inside of another block.

DNS

Somewhat surprising to me was how easy it was to setup a new domain with all of this in place. I found a domain name I liked, and set a single A-type record (since I still don't have much ability to use IPv6) with a "name" of @. With some time for DNS to propagate, I was setup and ready to go. I launched my two servers on their respective localhost ports and everything worked as intended.

Secondary HTTP Sites

With the "hard" part done, I wanted to setup a second HTTP site within Nginx for the new domain. All that required was a new file symlinked from /etc/nginx/sites-available/ to /etc/nginx/sites-enabled/ like this:

server {
  listen 80;
  server_name foo.com;
  root /var/www/foo.com/;
}

And I was setup for static file serving over HTTP too.

Thinking About Alternatives

I was a little surprised to find that Nginx handles this particular use-case, and talking to a few people they were similarly surprised. I don't think it is a bad thing, but it got me thinking about other ways of doing it so I started thumbing my way through Httpd and Relayd Mastery. I have idly thought about switching my own server(s) to OpenBSD in the past and this was another such instance. There is a clear separation between the web server (httpd), which is intentionally limited in terms of features and the business of routing, proxying, TLS termination and otherwise fiddling with network traffic. I was amused to learn that relayd itself only communicates with pf (the packet filter) for layer 3 redirection.

I haven't decided on anything, because everything continues to basically "just work" here on Debian but it's like a siren song, I'll probably have to check it out at some point.