What are we doing here?

Let's set up HAProxy with some lovely free certs from Let's Encrypt via certbot for a couple of domains. Everything running in docker, and all tied together with docker-compose. We'll use docker user-defined networks, because that's the Right Thing To Do.

I'll use 'domain1.example.com' and 'domain2.example.com' as the domains involved. The domain1 site is served from a container called 'container1', and domain2 from a container called 'domain2' You might have more. Or less. Edit as appropriate.

This wiki is set up very similarly to the below - the running config is on my github.

This should be easy. Right?

Docker: easy. HAProxy: easy. Let's Encrypt: easy.

Docker and HAProxy and Let's Encrypt: pain in the arse.

There's a few things that make this a bit of a hassle:

  1. We want haproxy to be running on port 80/443, but those are the ports certbot needs to do validation
    We'll have to do this in two stages.
  2. haproxy with the default config won't start up if it can't resolve the container IPs for the backends.
    Since certbot is just a command to be run in a container, it probably won't be running when haproxy starts up.
    Some extra config is needed in haproxy.
  3. certbot needs to be run once in one way to request the certs, and then every couple of days/weeks in another way to check and renew certs.
    We'll need two different incantations for certbot.
  4. When the certs are renewed, we'll need to tell haproxy to pick them up
    Some docker-in-docker magic is required.
  5. certbot doesn't know how to make haproxy-complicit cert pem files
    We'll need to do a little scripting.

Stage 0 - setup

This all assumes that your soon-to-be certificated domains' A records are all pointing at the docker host (or port-forwarding router, or whatever), and you can reach the docker host on port 80 and 443 from the Interwebs. Your docker host should have docker and docker-compose installed, and docker running.

Stage 1 - get some certs

Since this is a greenfield setup, we can let certbot take care of the initial cert request on its own - HAProxy should be down for this.

Dockerfile

The Dockerfile for the letsencrypt image looks like:

FROM ubuntu:latest

ENV DEBIAN_FRONTEND=noninteractive 
RUN apt-get update && \
    apt-get install -y software-properties-common && \
    add-apt-repository ppa:certbot/certbot && \
    apt-get update && \
    apt-get install -y certbot docker.io
COPY deploy-hook /deploy-hook
RUN chmod +x /deploy-hook

Note we're installing the docker.io package, and copying in a script. We'll need them later on.

deploy-hook

The deploy-hook script looks like:

#!/usr/bin/env bash

cat /etc/letsencrypt/live/domain1.example.com/fullchain.pem \
    /etc/letsencrypt/live/domain1.example.com/privkey.pem \
    > /etc/letsencrypt/haproxy.pem \
&& docker kill -s HUP haproxy

docker-compose-stage1.yml

To run the container, we'll wrap it up in a docker-compose file called docker-compose-stage1.yml.

version: '3'
  letsencrypt:
    build: .
    image: letsencrypt
    container_name: letsencrypt
    restart: no
    volumes:
      - letsencrypt_etc:/etc/letsencrypt
    command: bash -c 'certbot certonly \
                        --standalone \
                        --preferred-challenges http-01 \
                        --http-01-port 8000 \
                        --agree-tos \
                        --non-interactive \
                        -m your.email@fastmail.com \
                        -d "domain1.example.com" \
                        -d "domain2.example.com"; \
                      cat /etc/letsencrypt/live/domain1.example.com/fullchain.pem \
                        /etc/letsencrypt/live/domain1.example.com/privkey.pem \
                        > /etc/letsencrypt/haproxy.pem'
    ports:
      - 80:8000
    
volumes:
  letsencrypt_etc:

Things of note:

  1. certbot listens on port 8000, which docker is mapping to port 80 and making available to the outside world for Let's Encrypt to talk to. We don't need port 443 mapped, because this is an initial request and Let's Encrypt should be fine with just port 80.
  2. We're attaching a Volume to /etc/letsencrypt - that's where the certs end up, and that's how we'll make them available to haproxy.
  3. The command concatenates the cert chain and private key into a format that haproxy understands, and dumps it out into the mounted /etc/letsencrypt volume.
  4. certbot names the certs for the first domain specified, so that ends up in all of the paths under /etc/letsencrypt. You might be able to change that, but see rule 1.

Go!

With all three files in your current directory, run: docker-compose -f docker-compose-stage1.yml up and you should hopefully see a message like the following after a couple of seconds:

IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:

Certs gotten, and stashed away in a docker volume. Happy days!

Stage 2 - haproxy

We've got ourselves some certs so it's time to fire up haproxy and enjoy all the HAing and Proxying. Don't know about you, but I am excited.

Dockerfile

We don't need one. Becuase we're using the official image. Because we're adhering to rule 1.

haproxy.cfg

Do an mkdir haproxy/bind. Put something like the below in haproxy/bind/haproxy.fg:

global
    maxconn 4096
    daemon
    log stdout format raw local0 debug
    tune.ssl.default-dh-param 2048
    ssl-default-bind-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
    ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
    ssl-default-server-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
    ssl-default-server-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets

resolvers docker
    nameserver docker1 127.0.0.11:53

defaults
    log     global
    mode    http
    option  httplog
    timeout connect 5000
    timeout client  50000
    timeout server 50000
    default-server init-addr none

frontend http_in
    bind *:8080
    bind *:8443 ssl crt /etc/letsencrypt/haproxy.pem
    mode http
    redirect scheme https code 301 if !{ ssl_fc }

    capture request header Host len 256
    capture request header User-Agent len 256

    acl acme_pth path_beg -i /.well-known/acme-challenge
    acl domain1_hdr hdr(host) -i domain1.example.com
    acl domain2_hdr hdr(host) -i domain2.example.com

    use_backend letsencrypt if acme_pth
    use_backend domain1 if domain1_hdr
    use_backend domain2 if domain2_hdr

backend letsencrypt
    server letsencrypt letsencrypt:8000 resolvers docker check

backend domain1
    server domain1-1 container1:5000 resolvers docker check

backend wiki
    server domain2-1 container2:3000 resolvers docker check

What's going on here?

  1. The global section logs everything to stdout, because that's what you do with docker. rule 6 does not apply in dockerland.
  2. We're setting the Mozilla recommended ciphers and DH values. Check the current recommendations if you're foolish enough to go into production with this stuff.
  3. We're using 'resolvers' and 'default-server init-addr none' to get around problem of containers not being up at startup time. Docker with user-define networks always puts a resolver at 127.0.0.11:53, and haproxy can use that to resolve container names at runtime instead of startup time.
  4. We're binding to port 8080 and 8443, and setting the cert to the Let's Encrypt cert we dumped out in the previous section. Those ports will be mapped back to 80 and 443 by docker later on.
  5. Always redirect to https.
  6. All traffic that matches the certbot ACME protocol is directed to out letsencrypt container.
  7. Other traffic is matched by request hostname to their respective containers. Your routing will probably be more complicated than this, but it's a start.

Table of Contents