What are we doing here?

Let's set up HAProxy with some lovely free certs from Let's Encrypt via certbot for a couple of domains (or just one, if you like), each domain served from a different container, and all in docker.

The rules:

  • Everything running in docker, and all tied together with docker-compose.
  • No k8s, no swarm, just one woman/man/other and one host/VM/other. Comfortable. Oldskool.
  • We'll use docker user-defined networks, because that's the Right Thing To Do here.

I'll use 'domain1.example.com' and 'domain2.example.com' as example domains. The domain1 site would be served from a container called 'container1', and domain2 from 'container2' You might have more, or less, but edit as appropriate with your own or delete as you see fit. You'll need real actual internet resolvable domains to run through this.

This wiki you're reading is set up very similarly to the below - the running config is on my home-network repo.

You can grab the files listed here from the example repo if you're short on time/patience.

This should be easy. Right?

Docker: easy. HAProxy: easy. Let's Encrypt: easy.

Docker and HAProxy and Let's Encrypt: pain in the arse.

There's a few things that make this a bit of a hassle:

  • We want haproxy to be running on port 80/443, but those are the ports certbot needs to do validation.
    We'll do this in two stages for minimum pain.
  • haproxy with the default config won't start up if it can't resolve the container IPs for the backends.
    This is a general problem with haproxy and containers. We'll do some config to make it work.
  • certbot needs to be run one way to request the certs, and then every couple of days/weeks another way to check and renew certs.
    We'll need two different incantations for certbot.
  • When the certs are renewed, we'll need to tell haproxy to pick them up
    Some volumes and docker-in-docker magic is required.
  • certbot doesn't know how to make haproxy-complicit cert pem files
    We'll need to do a little scripting. Not much though. 3-lines, max.

Let's do this thing.

Stage 0 - setup

This all assumes that your soon-to-be certificated domains' A records are all pointing at the docker host (or port-forwarding router, or whatever), and you can reach the docker host on port 80 and 443 from the Interwebs. Your docker host should have docker, docker-compose, and openssl installed (openssl just for testing), and the docker daemon should be running. You should be root, or a similarly permissioned user.

Stage 1 - certbot

Since this is a greenfield setup, we can let certbot take care of the initial cert request on its own - nothing should be using ports 80 or 443 on the docker host at the moment so certbot can listen by itself. Create a letsencrypt directory to stuff this in, in your current directory.

Dockerfile

The letsencrypt/Dockerfile file looks like:

FROM ubuntu:latest

ENV DEBIAN_FRONTEND=noninteractive 
RUN apt-get update && \
    apt-get install -y software-properties-common && \
    add-apt-repository ppa:certbot/certbot && \
    apt-get update && \
    apt-get install -y certbot docker.io
COPY deploy-hook /deploy-hook
RUN chmod +x /deploy-hook

Note we're installing the docker.io package, and copying in a 'deploy-hook' script (see below). We'll need them later on. We could probably use the official certbot image, but chances are you'll already have 'ubuntu:latest' in-cache, so we might as well use it.

docker-compose-stage1.yml

To run the container, we'll wrap it up in a docker-compose file called docker-compose-stage1.yml. Put this in your current directory:

version: '3'

services:
  letsencrypt:
    build: ./letsencrypt
    image: letsencrypt
    container_name: letsencrypt
    restart: "no"
    volumes:
      - letsencrypt_etc:/etc/letsencrypt
    command: bash -c 'certbot certonly \
                        --standalone \
                        --preferred-challenges http-01 \
                        --http-01-port 8000 \
                        --agree-tos \
                        --non-interactive \
                        -m your.email@fastmail.com \
                        -d "domain1.example.com" \
                        -d "domain2.example.com"; \
                      cat /etc/letsencrypt/live/domain1.example.com/fullchain.pem \
                        /etc/letsencrypt/live/domain1.example.com/privkey.pem \
                        > /etc/letsencrypt/haproxy.pem'
    ports:
      - 80:8000
    
volumes:
  letsencrypt_etc:

Things of note:

  • certbot listens on port 8000, which docker is mapping to port 80 and making available to the outside world for Let's Encrypt to talk to. We don't need port 443 mapped, because this is an initial request and Let's Encrypt should be fine with just port 80.
  • We're attaching a docker volume to /etc/letsencrypt - that's where the certs end up, and that's how we'll make them available to haproxy.
  • The command concatenates the cert chain and private key into a format that haproxy understands, and dumps it out into the mounted /etc/letsencrypt volume.
  • certbot names the certs for the first domain specified, so that domain ends up in all of the paths under /etc/letsencrypt. You might be able to change that, but see rule 1.

deploy-hook

The letsencrypt/deploy-hook script looks like:

#!/usr/bin/env bash

cat /etc/letsencrypt/live/domain1.example.com/fullchain.pem \
    /etc/letsencrypt/live/domain1.example.com/privkey.pem \
    > /etc/letsencrypt/haproxy.pem \
&& docker kill -s HUP haproxy
Again, the first domain in the paths there.

Go!

Run: docker-compose -f docker-compose-stage1.yml up and you should hopefully see a message like the following after a couple of seconds:

IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:

Certs gotten, and stashed away in a docker volume. Happy days!

Stage 2 - haproxy

We've got ourselves some certs so it's time to fire up haproxy and enjoy all the HAing and Proxying. Don't know about you, but I'm suitably excited.

Dockerfile

We don't need one. Because we're using the official image. Because we're adhering to rule 1.

haproxy.cfg

Do an mkdir -p haproxy/bind. Put something like the below in haproxy/bind/haproxy.cfg:

global
    maxconn 4096
    daemon
    log stdout format raw local0 debug
    tune.ssl.default-dh-param 2048
    ssl-default-bind-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
    ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
    ssl-default-server-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
    ssl-default-server-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets

resolvers docker
    nameserver docker1 127.0.0.11:53

defaults
    log     global
    mode    http
    option  httplog
    timeout connect 5000
    timeout client  50000
    timeout server 50000
    default-server init-addr none

frontend http_in
    bind *:8080
    bind *:8443 ssl crt /etc/letsencrypt/haproxy.pem
    mode http
    redirect scheme https code 301 if !{ ssl_fc }

    capture request header Host len 256
    capture request header User-Agent len 256

    acl acme_pth path_beg -i /.well-known/acme-challenge
    acl domain1_hdr hdr(host) -i domain1.example.com
    acl domain2_hdr hdr(host) -i domain2.example.com

    use_backend letsencrypt if acme_pth
    use_backend domain1 if domain1_hdr
    use_backend domain2 if domain2_hdr

backend letsencrypt
    server letsencrypt letsencrypt:8000 resolvers docker check

backend domain1
    server domain1-1 container1:5000 resolvers docker check

backend domain2
    server domain2-1 container2:3000 resolvers docker check

What's going on here then?

  • The global section logs everything to stdout, because that's what you do with docker. rule 6 does not apply in dockerland.
  • We're setting the Mozilla recommended ciphers and DH values. Check the current recommendations if you're foolish enough to go into production with this stuff.
  • We're using 'resolvers' and 'default-server init-addr none' to get around the problem of containers not being up at haproxy startup time. Docker with user-defined networks always puts a resolver at 127.0.0.11:53, and haproxy can use that to resolve container names at runtime instead of startup time.
  • We're binding to port 8080 and 8443, and setting the cert to the Let's Encrypt cert we dumped out in the previous section. The ports will be mapped back to 80 and 443 by docker later on.
  • Always redirect to https.
  • All traffic that matches the certbot ACME challenge protocol is directed to our letsencrypt container (to be created later).
  • Other traffic is matched by request hostname to their respective containers. Your routing will probably be more complicated than this, but it's a start.

Let's wrap this up in docker-compose…

docker-compose.yml

In your current directory, put this is docker-compose.yml:

version: '3'

services:
  haproxy:
    container_name: haproxy
    image: haproxy:latest
    restart: always
    volumes:
      - ./haproxy/bind:/usr/local/etc/haproxy:ro,Z
      - letsencrypt_etc:/etc/letsencrypt:ro
    networks:
      - haproxy
    ports:
      - 80:8080
      - 443:8443
    user: '1001'
    
volumes:
  letsencrypt_etc:
    
networks:
  haproxy:
Here we have:
  • The container_name is 'haproxy'. We'll be referring to this container name later on for sending signals when certs are renewed (referenced in the deploy-hook script from stage 1).
  • The 'haproxy/bind' dir is mounted at /usr/local/etc/haproxy, so the haproxy.cfg file we created is in the right place for haproxy to read it. Mounted read-only, and with the 'Z' selinux flag (I'm running RedHat-ish host OSes here, so it's required - leave off the ',Z' if docker complains).
  • The letsencrypt volume is mounted at /etc/letsencrypt so haproxy can read the cert file.
  • We're creating a user-defined network called 'haproxy' so we can talk to other containers and have built-in dns work.
  • The high port numbers are mapped down to the usual 80/443 .
  • We're setting a non-priv UID to run as. Because containers don't need to run as root.

Go!

Run that bad boy with docker-compose up. You should see some startup messages and hopefully no errors. haproxy might complain about the backends being down, but that's OK for now.

Test port 443 from the docker host with: openssl s_client -connect localhost:443 | openssl x509 -text and you should see your cert if all has gone well.

Bring haproxy back down with docker-compose stop so we've got a clean slate for the next stage.

Stage 3 - automatic cert renewal

So far we've got haproxy up, with certs, and everything is just tickety boo. Those certs only last for 90 days though, and we're not in the habit of breaking rule 7. We'll need a container that can:

  • See the certificates we already have.
  • Renew them.
  • Tell haproxy something has changed.
  • Keep doing the above.

Dockerfile

We've already built the image for this in stage 1, so we're good to go.

docker-compose.yml

Add a letsencrypt stanza to the existing docker-compose.yml, so it looks like: 

version: '3'

services:
  haproxy:
    container_name: haproxy
    image: haproxy:latest
    restart: always
    volumes:
      - ./haproxy/bind:/usr/local/etc/haproxy:ro,Z
      - letsencrypt_etc:/etc/letsencrypt
    networks:
      - haproxy
    ports:
      - 80:8080
      - 443:8443
    user: '1001'
		
  letsencrypt:
    build: ./letsencrypt
    image: letsencrypt
    container_name: letsencrypt
    restart: always
    volumes:
      - letsencrypt_etc:/etc/letsencrypt
      - /var/run/docker.sock:/var/run/docker.sock:rw,Z
    networks:
      - haproxy
    command: bash -c 'sleep 10; while true; do certbot renew --deploy-hook /deploy-hook; sleep 86400; done'
    privileged: true
    
volumes:
  letsencrypt_etc:
    
networks:
  haproxy:
What doing?
  • We're mounting the letsencrypt volume back up at /etc/letsencrypt so 'certbot –renew' can operate on the certs.
  • The docker socket from the host is mounted at /var/run/docker.sock. This lets us do docker operations from inside the container.
  • There's a small sleep to let haproxy start up (ewww, but also, whatever), then we attempt a renew and run the deploy-hook script (see stage 1) if anything changed.
  • The deploy-hook script concatenates the cert chain and key into an haproxy style .pem file, then sends a SIGHUP via the docker command to the haproxy container, telling haproxy to re-read its config and pick up the new certs
  • The container is granted privileged permissions to let the docker socket work.

Go!

Run docker-compose up to bring up haproxy and the letsencrypt container. certbot will (after 10 seconds) read the current certs and decide there's nothing to do, then go to sleep for a day. haproxy should start up and tell you nice things about the letsencrypt backend being available.

Next?

Add your choice of backends and containers to haproxy.cfg and docker-compose.yml, and go about your business.

Relax

Table of Contents