First Attempt

The title of this section should give you an indication that I had some problems with this setup. Looking back, the biggest problem I had was that I really didn’t fully understand the underlying concepts. I’m pretty sure that if I went back now I could get my reverse proxy configured properly. On the other hand, I’m very happy with the current setup and would likely have migrated to this at some point anyway.

So first off, what exactly is a reverse proxy, and why do I need one? Let’s start off with what a proxy (or forward proxy) is; very simply, a forward proxy is a site that forwards your browser’s internet requests to the requested site (instead of your browser reaching out directly to the requested site). These are commonly used in businesses and schools because in addition to the forwarding aspect, the requests can also be intercepted and blocked.

A reverse proxy works the other way around. Instead of managing outbound requests, the reverse proxy handles inbound requests. Let’s say that you’re hosting a website inside your local network in a docker container called “mywebsite” on a host called “server.flora.family”. But you actually call your website “homebase.flora.family”. A reverse proxy can take that request and forward it to the correct server and docker container. For more complicated setups, they can also be configured to load balance if you have multiple servers all hosting the same website to help reduce the load on any one server. Personally, I won’t be taking advantage of the load balancing part until I can get some more hardware in my HomeLab, but being able to redirect URL’s to particular servers and containers is exactly what I needed.

The tricky part about reverse proxies for website is security; we need to make sure that we have a secure connection, so we need certificates. This means that a reverse proxy that works well with certificates, especially via Let’s Encrypt, is essential.

My first foray into reverse proxies involved the HAProxy plugin available for my firewall OPNsense. It seemed to check the boxes; it worked with the Let’s Encrypt plugin and I found several tutorials online detailing how to set it up. My problem was that I just could not get it working how I wanted it to; I know for a fact that this was a failure on my part, not on HAProxy’s part, but I just could not figure out what was wrong with my configuration. I was able (finally) to get it working with my first docker container, but when I went in and tried to set it up with the second I encountered more issues and was never able to have more than one container working at a time. This is something that I really wanted to work, but I was just having a really difficult time wrapping my head around it. I finally invoked my “Fun Time” goal; it was getting too frustrating. I started a new search to see how other people were running reverse proxies.

Securing Containers Behind a Reverse Proxy

My search almost immediately led me to a service called Traefik. This is a reverse proxy with integrated Let’s Encrypt support that integrates natively with Docker containers Docker swarms. Even better - once it’s up and running, all of the Traefik setup necessary is handled through the docker-compose file for your stacks. It’s run from a docker container itself, integrates with a bunch of different metric collection apps (such as Prometheus), and has a ton of other great features such as mirroring, canary deployments, and middlewares to customize the URL handling. It looked great!

The biggest issue I ran into with my Traefik (pronounced just like traffic) was that they recently released a 2.0 version that broke compatibility with their 1.0 version. They’ve got pretty good documentation on the 2.0 version, but the vast majority of user posts about Traefik are based on their 1.0 version. It took me a while to be able to recognize one from the other; the basic rule of thumb I found is that the docker-compose settings for 2.0 have much longer names with many more periods than their 1.0 counterparts.

Traefik 2.0 Configuration

I found installing Traefik 2.0 far easier than my attempts at installing HAProxy. But prerequisite to this was to have docker.io and docker-compose installed. Traefik uses the docker-compose configuration files to get the variables it needs.

NOTE: I keep all of my docker-compose files in separate directories on a ZFS dataset /library/config/. Please make sure to update that with whatever path you’re using.

Here’s my docker-compose configuration:

/library/config/traefik/docker-compose.yml
version: '3'

services:
  traefik:
    image: traefik
    container_name: traefik
    restart: unless-stopped
    security_opt:
      - no-new-privileges:true
    networks:
      - proxy-web
    ports:
      - 80:80
      - 443:443
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /library/config/traefik/traefik.yml:/traefik.yml:ro
      - /library/config/traefik/config:/config:ro
      - /library/config/traefik/data/acme.json:/acme.json
      - /library/config/traefik/logs/:/logs/
    labels:
      - traefik.enable=true
      - traefik.http.routers.traefik.entrypoints=web-80
      - traefik.http.routers.traefik.rule=Host(`traefik.flora.family`)
      - traefik.http.routers.traefik.middlewares=redirect-to-https@file
      - traefik.http.middlewares.test-auth.basicauth.users=***REDACTED***
      - traefik.http.routers.traefik-secure.entrypoints=web-443
      - traefik.http.routers.traefik-secure.rule=Host(`traefik.flora.family`)
      - traefik.http.routers.traefik-secure.middlewares=test-auth
      - traefik.http.routers.traefik-secure.tls=true
      - traefik.http.routers.traefik-secure.tls.certresolver=letsencrypt
      - traefik.http.routers.traefik-secure.service=api@internal

networks:
  proxy-web:
    external: true

Most of these sections are pretty standard for a docker-compose file. The networks: section at the bottom identifies an external docker network that all of the reverse-proxied containers will also subscribe to; this network needs to be defined with a separate docker command:

docker network create proxy-web

And then the networks: section in the Traefik definition subscribes to that network. The ports: section maps ports 80 and 443 (HTTP and HTTPS) from the host server to the Traefik container, and the volumes: section maps files and directories on the host server to directories in the container. localtime is a simple way to ensure the host and container use the same time and timezone and docker.sock provides Traefik with the info it needs to connect to the Docker API. The remainder map Traefik’s configuration and logs to my ZFS filesystem. Obviously, you’ll need to change the file locations where you want to store this information (to the left of the semi-colon). You’ll need to create the directory structure, but docker will create the files.

The labels: section is where you configure a container’s reverse proxy setup, and you’ll use labels very similar to these for any new docker-compose stack you create. Let’s go right down the list here.

• traefik.enable=true - This tells Traefik that it will be working with this container
• traefik.http.routers.traefik.entrypoints=web-80 - Defines a router called traefik that will use entrypoint web-80 (defined in the traefik.yml file, see below)
• traefik.http.routers.traefik.rule=Host(`traefik.flora.family`) - Defines a rule for the traefik router to listen for URL traefik.flora.family. Remember to change this to match your domain!
• traefik.http.routers.traefik.middlewares=redirect-to-https@file - Instructs traffic on the traefik router to follow the redirect-to-https middleware (defined in the file config/middleware.yml, see below)
• traefik.http.middlewares.test-auth.basicauth.users=***REDACTED*** - Defines a basicauth middleware titled test-auth, with the username and encrypted password redacted (see below)
• traefik.http.routers.traefik-secure.entrypoints=web-443 - Defines a new router titled traefik-secure that will use entrypoint web-443 (defined in the traefik.yml file, see below)
• traefik.http.routers.traefik-secure.rule=Host(`traefik.flora.family`) - Defines a rule for the traefik-secure router to listen for URL traefik.flora.family Remember to change this to match your domain!
• traefik.http.routers.traefik-secure.middlewares=test-auth - Instructs traffic on the traefik-secure router to follow the test-auth middleware (ie, ask for the user/pass specified above)
• traefik.http.routers.traefik-secure.tls=true - Specifies that TLS encryption is enabled on the routher traefik-secure
• traefik.http.routers.traefik-secure.tls.certresolver=letsencrypt - Specifies that the traefik-secure router will use the letsencrypt certificate resolver (defined in the traefik.yml file, see below)
• traefik.http.routers.traefik-secure.service=api@internal - Points traffic on the traefik-secure router to the internal Traefik Dashboard. For other containers, this would typically be the docker container_name as defined above.

The ***REDACTED*** part of the basicauth middleware definition above is the username and encrypted password for the user (or users) that have access to the Traefik dashboard. You can encrypt your password with the htpasswd command:

htpasswd -nb username secure_password

This will return something that looks like this:

username:oTt1/$ruca84Hq$m.DbMZBAG.KWn7vfN/SNK/

Copy/paste that entire line and replace the ***REDACTED*** text above.

Now we’re using a few names defined in other files. For the most part, this is just something that makes the configuration cleaner or more secure (especially for things we’ll be reusing regularly, like the Let’s Encrypt setup). So let’s take a look at the two files referenced above:

/library/config/traefik/traefik.yml
global:
  # Send anonymous usage data to the Traefik devs
  sendAnonymousUsage: true

api:
  dashboard: true

entryPoints:
  web-80:
    address: ":80"
  web-443:
    address: ":443"

providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false
  file:
    directory: /config
    watch: true

certificatesResolvers:
  letsencrypt:
    acme:
      email: christopher@flora.family
      storage: acme.json
      # Uncomment for staging server, comment for prod
      # caServer: "https://acme-staging-v02.api.letsencrypt.org/directory"
      tlsChallenge: {}

# Writing Logs to a File
log:
  filePath: "/logs/traefik.log"
  level: INFO

accessLog:
  filePath: "/logs/access.log"
  bufferingSize: 100
  filters:
    statusCodes:
      - "400-499"
      - "500-599"
    retryAttempts: true
#    minDuration: "10ms"

This file contains the global configuration for Traefik along with a few definitions for use in our docker configuration. Let’s start at the top.

sendAnonymousUsage: true - I don’t know about you, but when I’m getting a fantastic product for free I try to do what I can to give back. If my anonymized usage data helps the Traefik devs, then it’s the least I can do.

dashboard: true - this enabled the api@internal service we mentioned above. This gives you a dashboard that is tremendously helpful for troubleshooting. Just make sure you’re securing it properly with TLS certificates.

entrypoints: - here’s where we define the entrypoints used in the docker configuration. A lot of tutorials I saw name these http and https, but that seemed needlessly confusing to me. I named them this way so I could easily distinguish them from other uses of http in the configs.

providers: - the docker provider is necessary for Traefik to talk to Docker. The file provider defines a config directory for creating additional rules. Traefik will actively monitor this directory and pull in any new files or updates on the fly.

certificateResolvers: - this is all the configuration necessary for Traefik to get Let’s Encrypt certificates for your containers. Note that the acme.json file is defined in the volumes: section of the Traefik docker-compose file and should have only 600 level privileges.

log: and accessLog: - exactly what they look like; I set the accessLog file to only log HTTP 400 and 500 errors (no need to see the successes).

/library/config/traefik/config/middleware.yml
http:
  middlewares:
    redirect-to-https:
      redirectScheme:
        scheme: https
        permanent: true

Here’s the definition of the redirect-to-https middleware which will automatically redirect any traffic on our traefik router to our traefik-secure router.

NOTE: I said earlier how I hate it when tutorials use the same words in different contexts as it’s needlessly confusing. Well, I did it here; the word traefik in this sense is in no way a special keyword, I could have called the routers blerg instead of traefik and blergawerg instead of traefik-secure. I did at least use the same nomenclature for all my services (eg Portainer has a portainer and portainer-secure router). The main take-away is that the fourth word in these router labels is the name of the router. So traefik.http.router.blerg.rule= refers to the router named blerg.

Traefik 2.0 Installation

After all of that work, the installation is easy! From the directory where your docker-compose file lives, all you have to do is run:

docker-compose up -d

The -d means to run in detached mode (ie, in the background). If you’re having trouble, try rerunning the command above without the -d.

Now if you point your browser to the URL you defined above (mine is at traefik.flora.family) you should see your Traefik dashboard!

Wrapping Up

The next step is to set up docker containers, and they’re done the same way as we’ve done here with Traefik. Get your docker-compose file for whatever else you want to run, and use these same labels, slightly modified for your new service.

In the next post in this series, I’ll show how to set up Portainer a fairly lightweight self-hosted service to help manage your Docker stacks.

In the meantime, please reach out to me at christopher@flora.family if you have any questions!