d3077669ece33bae42d7b6ac599b8039487f15db
compute/haproxy-letsencrypt-docker.md
| ... | ... | @@ -0,0 +1,341 @@ |
| 1 | +<!-- TITLE: HAProxy with Lets Encrypt in Docker --> |
|
| 2 | +<!-- SUBTITLE: This is 3 lines of bash when docker's not involved you know --> |
|
| 3 | + |
|
| 4 | +# What are we doing here? |
|
| 5 | +Let's set up HAProxy with some lovely free certs from [Let's Encrypt](https://letsencrypt.org/) via [certbot](https://certbot.eff.org/) for a couple of domains (or just one, if you like), each domain served from a different container, and all in docker. |
|
| 6 | + |
|
| 7 | +The rules: |
|
| 8 | +* Everything running in docker, and all tied together with docker-compose. |
|
| 9 | +* No k8s, no swarm, just one woman/man/other and one host/VM/other. Comfortable. Oldskool. |
|
| 10 | +* We'll use [docker user-defined networks](https://docs.docker.com/v17.09/engine/userguide/networking/#user-defined-networks), because that's the Right Thing To Do here. |
|
| 11 | + |
|
| 12 | +I'll use 'domain1.example.com' and 'domain2.example.com' as example domains. The domain1 site would be served from a container called 'container1', and domain2 from 'container2' You might have more, or less, but edit as appropriate with your own or delete as you see fit. You'll need real actual internet resolvable domains to run through this though. |
|
| 13 | + |
|
| 14 | +This wiki you're reading is set up very similarly to the below - the running config is on [my home-network repo](https://github.com/ilikejam/home-network). |
|
| 15 | + |
|
| 16 | +You can grab the files listed here from [the example repo](https://github.com/ilikejam/haproxy-le-docker) if you're short on time/patience. |
|
| 17 | + |
|
| 18 | +# This should be easy. Right? |
|
| 19 | +Docker: easy. |
|
| 20 | +HAProxy: easy. |
|
| 21 | +Let's Encrypt: easy. |
|
| 22 | + |
|
| 23 | +Docker and HAProxy and Let's Encrypt: *pain in the arse*. |
|
| 24 | + |
|
| 25 | +There's a few things that make this a bit of a hassle: |
|
| 26 | +* We want haproxy to be running on port 80/443, but those are the ports certbot needs to do validation.<br/>We'll do this in two stages for minimum pain. |
|
| 27 | +* haproxy with the default config won't start up if it can't resolve the container IPs for the backends.<br/>This is a general problem with haproxy and containers. We'll do some config to make it work. |
|
| 28 | +* certbot needs to be run one way to request the certs, and then every couple of days/weeks another way to check and renew certs.<br/>We'll need two different incantations for certbot. |
|
| 29 | +* When the certs are renewed, we'll need to tell haproxy to pick them up<br/>Some volumes and docker socket magic is required. |
|
| 30 | +* certbot doesn't know how to make haproxy-complicit cert pem files<br/>We'll need to do a little scripting. Not much though. 3-lines, max. |
|
| 31 | + |
|
| 32 | +Let's do this thing. |
|
| 33 | + |
|
| 34 | +# Stage 0 - setup |
|
| 35 | +This all assumes that your soon-to-be certificated domains' A records are all pointing at the docker host (or port-forwarding router, or whatever), and you can reach the docker host on port 80 and 443 from the Interwebs. |
|
| 36 | +Your docker host should have docker, docker-compose, and openssl installed (openssl just for testing), and the docker daemon should be running. |
|
| 37 | +You should be root, or a similarly permissioned user. |
|
| 38 | + |
|
| 39 | +# Stage 1 - certbot |
|
| 40 | +Since this is a greenfield setup, we can let certbot take care of the initial cert request on its own - nothing should be using ports 80 or 443 on the docker host at the moment so certbot can listen by itself. |
|
| 41 | +Create a `letsencrypt` directory to stuff this in, in your current directory. |
|
| 42 | + |
|
| 43 | +## Dockerfile |
|
| 44 | +The `letsencrypt/Dockerfile` file looks like: |
|
| 45 | + |
|
| 46 | +```dockerfile |
|
| 47 | +FROM ubuntu:18.04 |
|
| 48 | + |
|
| 49 | +ENV DEBIAN_FRONTEND=noninteractive |
|
| 50 | +RUN apt-get update && \ |
|
| 51 | + apt-get install -y software-properties-common && \ |
|
| 52 | + add-apt-repository ppa:certbot/certbot && \ |
|
| 53 | + apt-get update && \ |
|
| 54 | + apt-get install -y certbot docker.io |
|
| 55 | +COPY deploy-hook /deploy-hook |
|
| 56 | +RUN chmod +x /deploy-hook |
|
| 57 | +``` |
|
| 58 | + |
|
| 59 | +Note we're installing the docker.io package, and copying in a 'deploy-hook' script (see below). We'll need them later on. We could probably use the official certbot image, but chances are you'll already have 'ubuntu:18.04' in-cache, so we might as well use it. |
|
| 60 | + |
|
| 61 | +## docker-compose-stage1.yml |
|
| 62 | +To run the container, we'll wrap it up in a docker-compose file called `docker-compose-stage1.yml`. Put this in your current directory: |
|
| 63 | + |
|
| 64 | +```yaml |
|
| 65 | +version: '3' |
|
| 66 | + |
|
| 67 | +services: |
|
| 68 | + letsencrypt: |
|
| 69 | + build: ./letsencrypt |
|
| 70 | + image: letsencrypt |
|
| 71 | + container_name: letsencrypt |
|
| 72 | + restart: "no" |
|
| 73 | + volumes: |
|
| 74 | + - letsencrypt_etc:/etc/letsencrypt |
|
| 75 | + command: bash -c 'certbot certonly \ |
|
| 76 | + --standalone \ |
|
| 77 | + --preferred-challenges http-01 \ |
|
| 78 | + --http-01-port 8000 \ |
|
| 79 | + --agree-tos \ |
|
| 80 | + --non-interactive \ |
|
| 81 | + -m your.email@fastmail.com \ |
|
| 82 | + -d "domain1.example.com" \ |
|
| 83 | + -d "domain2.example.com"; \ |
|
| 84 | + cat /etc/letsencrypt/live/domain1.example.com/fullchain.pem \ |
|
| 85 | + /etc/letsencrypt/live/domain1.example.com/privkey.pem \ |
|
| 86 | + > /etc/letsencrypt/haproxy.pem' |
|
| 87 | + ports: |
|
| 88 | + - 80:8000 |
|
| 89 | + |
|
| 90 | +volumes: |
|
| 91 | + letsencrypt_etc: |
|
| 92 | +``` |
|
| 93 | + |
|
| 94 | +Things of note: |
|
| 95 | +* certbot listens on port 8000, which docker is mapping to port 80 and making available to the outside world for Let's Encrypt to talk to. We don't need port 443 mapped, because this is an initial request and Let's Encrypt should be fine with just port 80. |
|
| 96 | +* We're attaching a docker volume to /etc/letsencrypt - that's where the certs end up, and that's how we'll make them available to haproxy. |
|
| 97 | +* The command concatenates the cert chain and private key into a format that haproxy understands, and dumps it out into the mounted /etc/letsencrypt volume. |
|
| 98 | +* certbot names the certs for the first domain specified, so that domain ends up in all of the paths under /etc/letsencrypt. You might be able to change that, but see [rule 1](/rules#1-love-thy-defaults). |
|
| 99 | + |
|
| 100 | +## deploy-hook |
|
| 101 | +The `letsencrypt/deploy-hook` script looks like: |
|
| 102 | + |
|
| 103 | +```sh |
|
| 104 | +#!/usr/bin/env bash |
|
| 105 | + |
|
| 106 | +cat /etc/letsencrypt/live/domain1.example.com/fullchain.pem \ |
|
| 107 | + /etc/letsencrypt/live/domain1.example.com/privkey.pem \ |
|
| 108 | + > /etc/letsencrypt/haproxy.pem \ |
|
| 109 | +&& docker kill -s HUP haproxy |
|
| 110 | +``` |
|
| 111 | +Again, the first domain in the paths there. |
|
| 112 | + |
|
| 113 | +## Go! |
|
| 114 | +Run: `docker-compose -f docker-compose-stage1.yml up` and you should hopefully see a message like the following after a couple of seconds: |
|
| 115 | + |
|
| 116 | +```text |
|
| 117 | +IMPORTANT NOTES: |
|
| 118 | + - Congratulations! Your certificate and chain have been saved at: |
|
| 119 | +``` |
|
| 120 | + |
|
| 121 | +Certs gotten, and stashed away in a docker volume. Happy days! |
|
| 122 | + |
|
| 123 | +# Stage 2 - haproxy |
|
| 124 | +We've got ourselves some certs so it's time to fire up haproxy and enjoy all the HAing and Proxying. Exciting times. |
|
| 125 | + |
|
| 126 | +## Dockerfile |
|
| 127 | +We don't need one. |
|
| 128 | +Because we're using the official image. |
|
| 129 | +Because we're adhering to [rule 1](/rules#1-love-thy-defaults). |
|
| 130 | + |
|
| 131 | +## haproxy.cfg |
|
| 132 | +Do an `mkdir -p haproxy/bind`. |
|
| 133 | +Put something like the below in `haproxy/bind/haproxy.cfg`: |
|
| 134 | + |
|
| 135 | +```text |
|
| 136 | +global |
|
| 137 | + maxconn 4096 |
|
| 138 | + daemon |
|
| 139 | + log stdout format raw local0 debug |
|
| 140 | + tune.ssl.default-dh-param 2048 |
|
| 141 | + ssl-default-bind-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256 |
|
| 142 | + ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets |
|
| 143 | + ssl-default-server-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256 |
|
| 144 | + ssl-default-server-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets |
|
| 145 | + |
|
| 146 | +resolvers docker |
|
| 147 | + nameserver docker1 127.0.0.11:53 |
|
| 148 | + |
|
| 149 | +defaults |
|
| 150 | + log global |
|
| 151 | + mode http |
|
| 152 | + option httplog |
|
| 153 | + timeout connect 5000 |
|
| 154 | + timeout client 50000 |
|
| 155 | + timeout server 50000 |
|
| 156 | + default-server init-addr none |
|
| 157 | + |
|
| 158 | +frontend http_in |
|
| 159 | + bind *:8080 |
|
| 160 | + bind *:8443 ssl crt /etc/letsencrypt/haproxy.pem |
|
| 161 | + mode http |
|
| 162 | + |
|
| 163 | + capture request header Host len 256 |
|
| 164 | + capture request header User-Agent len 256 |
|
| 165 | + |
|
| 166 | + acl acme_pth path_beg -i /.well-known/acme-challenge |
|
| 167 | + acl domain1_hdr hdr(host) -i domain1.example.com |
|
| 168 | + acl domain2_hdr hdr(host) -i domain2.example.com |
|
| 169 | + |
|
| 170 | + redirect scheme https code 301 if !{ ssl_fc } !acme_pth |
|
| 171 | + |
|
| 172 | + use_backend letsencrypt if acme_pth |
|
| 173 | + use_backend domain1 if domain1_hdr |
|
| 174 | + use_backend domain2 if domain2_hdr |
|
| 175 | + |
|
| 176 | +backend letsencrypt |
|
| 177 | + server letsencrypt letsencrypt:8000 resolvers docker |
|
| 178 | + |
|
| 179 | +backend domain1 |
|
| 180 | + server domain1-1 container1:5000 resolvers docker check |
|
| 181 | + |
|
| 182 | +backend domain2 |
|
| 183 | + server domain2-1 container2:3000 resolvers docker check |
|
| 184 | +``` |
|
| 185 | + |
|
| 186 | +What's going on here then? |
|
| 187 | +* The global section logs everything to stdout, because that's what you do with docker. [rule 6](/rules#6-thou-shalt-respect-the-sanctity-of-stdout) does not apply in dockerland. |
|
| 188 | +* We're setting the Mozilla recommended ciphers and DH values. Check the [current recommendations](https://mozilla.github.io/server-side-tls/ssl-config-generator/) if you're mental enough to go into production with this stuff. |
|
| 189 | +* We're using 'resolvers' and 'default-server init-addr none' to get around the problem of containers not being up at haproxy startup time. Docker with user-defined networks always puts a resolver at 127.0.0.11:53, and haproxy can use that to resolve container names at *runtime* instead of *startup* time. |
|
| 190 | +* We're *not* running 'check' on the letsencrypt backend - it will be down most of the time, and we don't care. |
|
| 191 | +* We're binding to port 8080 and 8443, and setting the cert to the Let's Encrypt cert we dumped out in the previous section. The ports will be mapped back to 80 and 443 by docker later on. |
|
| 192 | +* Always redirect to https. |
|
| 193 | +* All traffic that matches the certbot [ACME](https://ietf-wg-acme.github.io/acme/draft-ietf-acme-acme.html) challenge protocol is directed to our letsencrypt container (to be created later). |
|
| 194 | +* Other traffic is matched by request hostname to their respective containers. Your routing will probably be more complicated than this, but it's a start. |
|
| 195 | + |
|
| 196 | +Let's wrap this up in docker-compose... |
|
| 197 | + |
|
| 198 | +## docker-compose.yml |
|
| 199 | +In your current directory, put this is `docker-compose.yml`: |
|
| 200 | + |
|
| 201 | +```yaml |
|
| 202 | +version: '3' |
|
| 203 | + |
|
| 204 | +services: |
|
| 205 | + haproxy: |
|
| 206 | + container_name: haproxy |
|
| 207 | + image: haproxy:2.0 |
|
| 208 | + restart: always |
|
| 209 | + volumes: |
|
| 210 | + - ./haproxy/bind:/usr/local/etc/haproxy:ro,Z |
|
| 211 | + - letsencrypt_etc:/etc/letsencrypt:ro |
|
| 212 | + networks: |
|
| 213 | + - haproxy |
|
| 214 | + ports: |
|
| 215 | + - 80:8080 |
|
| 216 | + - 443:8443 |
|
| 217 | + user: '1001' |
|
| 218 | + |
|
| 219 | +volumes: |
|
| 220 | + letsencrypt_etc: |
|
| 221 | + |
|
| 222 | +networks: |
|
| 223 | + haproxy: |
|
| 224 | +``` |
|
| 225 | +Here we have: |
|
| 226 | +* The container_name is 'haproxy'. We'll be referring to this container name later on for sending signals when certs are renewed (referenced in the deploy-hook script from stage 1). |
|
| 227 | +* The 'haproxy/bind' dir is mounted at /usr/local/etc/haproxy, so the haproxy.cfg file we created is in the right place for haproxy to read it. Mounted read-only, and with the 'Z' selinux flag (I'm running RedHat-ish host OSes here, so it's required - leave off the ',Z' if docker complains). |
|
| 228 | +* The letsencrypt volume is mounted at /etc/letsencrypt so haproxy can read the cert file. |
|
| 229 | +* We're creating a user-defined network called 'haproxy' so we can talk to other containers and have built-in dns work. |
|
| 230 | +* The high port numbers are mapped down to the usual 80/443 . |
|
| 231 | +* We're setting a non-priv UID to run as. Because [containers don't need to run as root](https://medium.com/@mccode/processes-in-containers-should-not-run-as-root-2feae3f0df3b). |
|
| 232 | + |
|
| 233 | +## Go! |
|
| 234 | +Run that bad boy with `docker-compose up`. You should see some startup messages and hopefully no errors. haproxy might complain about the backends being down, but that's OK for now. |
|
| 235 | + |
|
| 236 | +Test port 443 from the docker host with: |
|
| 237 | +`openssl s_client -connect localhost:443 | openssl x509 -text` |
|
| 238 | +and you should see your cert if all has gone well. |
|
| 239 | + |
|
| 240 | +Bring haproxy back down with `docker-compose stop` so we've got a clean slate for the next stage. |
|
| 241 | + |
|
| 242 | +# Stage 3 - automatic cert renewal |
|
| 243 | +So far we've got haproxy up, with certs, and everything is just [tickety boo](https://en.wiktionary.org/wiki/tickety-boo). |
|
| 244 | +Those certs only last for 90 days though, and we're not in the habit of breaking [rule 7](/rules#7-thou-shalt-automate-everything). We'll need a container that can: |
|
| 245 | +* See the certificates we already have. |
|
| 246 | +* Renew them. |
|
| 247 | +* Tell haproxy something has changed. |
|
| 248 | +* Keep doing the above. |
|
| 249 | + |
|
| 250 | +## Dockerfile |
|
| 251 | +We've already built the image for this in stage 1, so we're good to go. |
|
| 252 | + |
|
| 253 | +## docker-compose.yml |
|
| 254 | +Add a letsencrypt stanza to the existing `docker-compose.yml`, so it looks like: |
|
| 255 | +```yaml |
|
| 256 | +version: '3' |
|
| 257 | + |
|
| 258 | +services: |
|
| 259 | + haproxy: |
|
| 260 | + container_name: haproxy |
|
| 261 | + image: haproxy:2.0 |
|
| 262 | + restart: always |
|
| 263 | + volumes: |
|
| 264 | + - ./haproxy/bind:/usr/local/etc/haproxy:ro,Z |
|
| 265 | + - letsencrypt_etc:/etc/letsencrypt |
|
| 266 | + networks: |
|
| 267 | + - haproxy |
|
| 268 | + ports: |
|
| 269 | + - 80:8080 |
|
| 270 | + - 443:8443 |
|
| 271 | + user: '1001' |
|
| 272 | + |
|
| 273 | + letsencrypt: |
|
| 274 | + build: ./letsencrypt |
|
| 275 | + image: letsencrypt |
|
| 276 | + container_name: letsencrypt |
|
| 277 | + restart: always |
|
| 278 | + volumes: |
|
| 279 | + - letsencrypt_etc:/etc/letsencrypt |
|
| 280 | + - /var/run/docker.sock:/var/run/docker.sock:rw,Z |
|
| 281 | + networks: |
|
| 282 | + - haproxy |
|
| 283 | + command: bash -c 'sleep 10; while true; do certbot renew --standalone --http-01-port 8000 --deploy-hook /deploy-hook; sleep 86400; done' |
|
| 284 | + privileged: true |
|
| 285 | + |
|
| 286 | +volumes: |
|
| 287 | + letsencrypt_etc: |
|
| 288 | + |
|
| 289 | +networks: |
|
| 290 | + haproxy: |
|
| 291 | +``` |
|
| 292 | +What doing? |
|
| 293 | +* We're mounting the letsencrypt volume back up at /etc/letsencrypt so 'certbot --renew' can operate on the certs. |
|
| 294 | +* The docker socket from the host is mounted at /var/run/docker.sock. This lets us do docker operations from inside the container. |
|
| 295 | +* There's a small sleep to let haproxy start up (ewww, but also, whatever), then we attempt a renew and run the deploy-hook script (see stage 1) if anything changed. |
|
| 296 | +* The deploy-hook script concatenates the cert chain and key into an haproxy style .pem file, then sends a SIGHUP via the docker command to the haproxy container, telling haproxy to re-read its config and pick up the new certs |
|
| 297 | +* The container is granted privileged permissions to let the docker socket work. |
|
| 298 | + |
|
| 299 | +## Go! |
|
| 300 | +Run `docker-compose up` to bring up haproxy and the letsencrypt container. certbot will (after 10 seconds) read the current certs and decide there's nothing to do, then go to sleep for a day. |
|
| 301 | +haproxy should start up and tell you nice things about the letsencrypt backend being available. |
|
| 302 | + |
|
| 303 | +# Add new domains and containers |
|
| 304 | +You're on your own with the container, but haproxy and certbot config we can do. |
|
| 305 | + |
|
| 306 | +## DNS |
|
| 307 | +Make sure the new domain's A record is pointing at haproxy's IP. |
|
| 308 | + |
|
| 309 | +## haproxy |
|
| 310 | +Add the new domain to haproxy.conf: |
|
| 311 | +```text |
|
| 312 | +... |
|
| 313 | + acl logs_hdr hdr(host) -i new.domain.com |
|
| 314 | +... |
|
| 315 | + use_backend new-container if logs_hdr |
|
| 316 | +... |
|
| 317 | +backend new-container |
|
| 318 | + server new-container1 new-container:6666 resolvers docker check |
|
| 319 | +``` |
|
| 320 | + |
|
| 321 | +## Container |
|
| 322 | +Bring up the new container if it's not already running. |
|
| 323 | + |
|
| 324 | +## letsencrypt |
|
| 325 | +Log into the letsencrypt container: |
|
| 326 | +`docker exec -ti letsencrypt bash` |
|
| 327 | + |
|
| 328 | +Then run certbot with the new domain and run the hook script: |
|
| 329 | +```bash |
|
| 330 | +$ certbot --standalone --expand \ |
|
| 331 | +-d "domain1.example.com" \ |
|
| 332 | +-d "domain2.example.com" \ |
|
| 333 | +-d "new.domain.com" |
|
| 334 | +$ /deploy-hook |
|
| 335 | +``` |
|
| 336 | + |
|
| 337 | +# Next? |
|
| 338 | +Do what you like. I'm not the boss of you. |
|
| 339 | + |
|
| 340 | +# Relax |
|
| 341 | + |
haproxy-letsencrypt-docker.md
| ... | ... | @@ -1,341 +0,0 @@ |
| 1 | -<!-- TITLE: HAProxy with Lets Encrypt in Docker --> |
|
| 2 | -<!-- SUBTITLE: This is 3 lines of bash when docker's not involved you know --> |
|
| 3 | - |
|
| 4 | -# What are we doing here? |
|
| 5 | -Let's set up HAProxy with some lovely free certs from [Let's Encrypt](https://letsencrypt.org/) via [certbot](https://certbot.eff.org/) for a couple of domains (or just one, if you like), each domain served from a different container, and all in docker. |
|
| 6 | - |
|
| 7 | -The rules: |
|
| 8 | -* Everything running in docker, and all tied together with docker-compose. |
|
| 9 | -* No k8s, no swarm, just one woman/man/other and one host/VM/other. Comfortable. Oldskool. |
|
| 10 | -* We'll use [docker user-defined networks](https://docs.docker.com/v17.09/engine/userguide/networking/#user-defined-networks), because that's the Right Thing To Do here. |
|
| 11 | - |
|
| 12 | -I'll use 'domain1.example.com' and 'domain2.example.com' as example domains. The domain1 site would be served from a container called 'container1', and domain2 from 'container2' You might have more, or less, but edit as appropriate with your own or delete as you see fit. You'll need real actual internet resolvable domains to run through this though. |
|
| 13 | - |
|
| 14 | -This wiki you're reading is set up very similarly to the below - the running config is on [my home-network repo](https://github.com/ilikejam/home-network). |
|
| 15 | - |
|
| 16 | -You can grab the files listed here from [the example repo](https://github.com/ilikejam/haproxy-le-docker) if you're short on time/patience. |
|
| 17 | - |
|
| 18 | -# This should be easy. Right? |
|
| 19 | -Docker: easy. |
|
| 20 | -HAProxy: easy. |
|
| 21 | -Let's Encrypt: easy. |
|
| 22 | - |
|
| 23 | -Docker and HAProxy and Let's Encrypt: *pain in the arse*. |
|
| 24 | - |
|
| 25 | -There's a few things that make this a bit of a hassle: |
|
| 26 | -* We want haproxy to be running on port 80/443, but those are the ports certbot needs to do validation.<br/>We'll do this in two stages for minimum pain. |
|
| 27 | -* haproxy with the default config won't start up if it can't resolve the container IPs for the backends.<br/>This is a general problem with haproxy and containers. We'll do some config to make it work. |
|
| 28 | -* certbot needs to be run one way to request the certs, and then every couple of days/weeks another way to check and renew certs.<br/>We'll need two different incantations for certbot. |
|
| 29 | -* When the certs are renewed, we'll need to tell haproxy to pick them up<br/>Some volumes and docker socket magic is required. |
|
| 30 | -* certbot doesn't know how to make haproxy-complicit cert pem files<br/>We'll need to do a little scripting. Not much though. 3-lines, max. |
|
| 31 | - |
|
| 32 | -Let's do this thing. |
|
| 33 | - |
|
| 34 | -# Stage 0 - setup |
|
| 35 | -This all assumes that your soon-to-be certificated domains' A records are all pointing at the docker host (or port-forwarding router, or whatever), and you can reach the docker host on port 80 and 443 from the Interwebs. |
|
| 36 | -Your docker host should have docker, docker-compose, and openssl installed (openssl just for testing), and the docker daemon should be running. |
|
| 37 | -You should be root, or a similarly permissioned user. |
|
| 38 | - |
|
| 39 | -# Stage 1 - certbot |
|
| 40 | -Since this is a greenfield setup, we can let certbot take care of the initial cert request on its own - nothing should be using ports 80 or 443 on the docker host at the moment so certbot can listen by itself. |
|
| 41 | -Create a `letsencrypt` directory to stuff this in, in your current directory. |
|
| 42 | - |
|
| 43 | -## Dockerfile |
|
| 44 | -The `letsencrypt/Dockerfile` file looks like: |
|
| 45 | - |
|
| 46 | -```dockerfile |
|
| 47 | -FROM ubuntu:18.04 |
|
| 48 | - |
|
| 49 | -ENV DEBIAN_FRONTEND=noninteractive |
|
| 50 | -RUN apt-get update && \ |
|
| 51 | - apt-get install -y software-properties-common && \ |
|
| 52 | - add-apt-repository ppa:certbot/certbot && \ |
|
| 53 | - apt-get update && \ |
|
| 54 | - apt-get install -y certbot docker.io |
|
| 55 | -COPY deploy-hook /deploy-hook |
|
| 56 | -RUN chmod +x /deploy-hook |
|
| 57 | -``` |
|
| 58 | - |
|
| 59 | -Note we're installing the docker.io package, and copying in a 'deploy-hook' script (see below). We'll need them later on. We could probably use the official certbot image, but chances are you'll already have 'ubuntu:18.04' in-cache, so we might as well use it. |
|
| 60 | - |
|
| 61 | -## docker-compose-stage1.yml |
|
| 62 | -To run the container, we'll wrap it up in a docker-compose file called `docker-compose-stage1.yml`. Put this in your current directory: |
|
| 63 | - |
|
| 64 | -```yaml |
|
| 65 | -version: '3' |
|
| 66 | - |
|
| 67 | -services: |
|
| 68 | - letsencrypt: |
|
| 69 | - build: ./letsencrypt |
|
| 70 | - image: letsencrypt |
|
| 71 | - container_name: letsencrypt |
|
| 72 | - restart: "no" |
|
| 73 | - volumes: |
|
| 74 | - - letsencrypt_etc:/etc/letsencrypt |
|
| 75 | - command: bash -c 'certbot certonly \ |
|
| 76 | - --standalone \ |
|
| 77 | - --preferred-challenges http-01 \ |
|
| 78 | - --http-01-port 8000 \ |
|
| 79 | - --agree-tos \ |
|
| 80 | - --non-interactive \ |
|
| 81 | - -m your.email@fastmail.com \ |
|
| 82 | - -d "domain1.example.com" \ |
|
| 83 | - -d "domain2.example.com"; \ |
|
| 84 | - cat /etc/letsencrypt/live/domain1.example.com/fullchain.pem \ |
|
| 85 | - /etc/letsencrypt/live/domain1.example.com/privkey.pem \ |
|
| 86 | - > /etc/letsencrypt/haproxy.pem' |
|
| 87 | - ports: |
|
| 88 | - - 80:8000 |
|
| 89 | - |
|
| 90 | -volumes: |
|
| 91 | - letsencrypt_etc: |
|
| 92 | -``` |
|
| 93 | - |
|
| 94 | -Things of note: |
|
| 95 | -* certbot listens on port 8000, which docker is mapping to port 80 and making available to the outside world for Let's Encrypt to talk to. We don't need port 443 mapped, because this is an initial request and Let's Encrypt should be fine with just port 80. |
|
| 96 | -* We're attaching a docker volume to /etc/letsencrypt - that's where the certs end up, and that's how we'll make them available to haproxy. |
|
| 97 | -* The command concatenates the cert chain and private key into a format that haproxy understands, and dumps it out into the mounted /etc/letsencrypt volume. |
|
| 98 | -* certbot names the certs for the first domain specified, so that domain ends up in all of the paths under /etc/letsencrypt. You might be able to change that, but see [rule 1](/rules#1-love-thy-defaults). |
|
| 99 | - |
|
| 100 | -## deploy-hook |
|
| 101 | -The `letsencrypt/deploy-hook` script looks like: |
|
| 102 | - |
|
| 103 | -```sh |
|
| 104 | -#!/usr/bin/env bash |
|
| 105 | - |
|
| 106 | -cat /etc/letsencrypt/live/domain1.example.com/fullchain.pem \ |
|
| 107 | - /etc/letsencrypt/live/domain1.example.com/privkey.pem \ |
|
| 108 | - > /etc/letsencrypt/haproxy.pem \ |
|
| 109 | -&& docker kill -s HUP haproxy |
|
| 110 | -``` |
|
| 111 | -Again, the first domain in the paths there. |
|
| 112 | - |
|
| 113 | -## Go! |
|
| 114 | -Run: `docker-compose -f docker-compose-stage1.yml up` and you should hopefully see a message like the following after a couple of seconds: |
|
| 115 | - |
|
| 116 | -```text |
|
| 117 | -IMPORTANT NOTES: |
|
| 118 | - - Congratulations! Your certificate and chain have been saved at: |
|
| 119 | -``` |
|
| 120 | - |
|
| 121 | -Certs gotten, and stashed away in a docker volume. Happy days! |
|
| 122 | - |
|
| 123 | -# Stage 2 - haproxy |
|
| 124 | -We've got ourselves some certs so it's time to fire up haproxy and enjoy all the HAing and Proxying. Exciting times. |
|
| 125 | - |
|
| 126 | -## Dockerfile |
|
| 127 | -We don't need one. |
|
| 128 | -Because we're using the official image. |
|
| 129 | -Because we're adhering to [rule 1](/rules#1-love-thy-defaults). |
|
| 130 | - |
|
| 131 | -## haproxy.cfg |
|
| 132 | -Do an `mkdir -p haproxy/bind`. |
|
| 133 | -Put something like the below in `haproxy/bind/haproxy.cfg`: |
|
| 134 | - |
|
| 135 | -```text |
|
| 136 | -global |
|
| 137 | - maxconn 4096 |
|
| 138 | - daemon |
|
| 139 | - log stdout format raw local0 debug |
|
| 140 | - tune.ssl.default-dh-param 2048 |
|
| 141 | - ssl-default-bind-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256 |
|
| 142 | - ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets |
|
| 143 | - ssl-default-server-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256 |
|
| 144 | - ssl-default-server-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets |
|
| 145 | - |
|
| 146 | -resolvers docker |
|
| 147 | - nameserver docker1 127.0.0.11:53 |
|
| 148 | - |
|
| 149 | -defaults |
|
| 150 | - log global |
|
| 151 | - mode http |
|
| 152 | - option httplog |
|
| 153 | - timeout connect 5000 |
|
| 154 | - timeout client 50000 |
|
| 155 | - timeout server 50000 |
|
| 156 | - default-server init-addr none |
|
| 157 | - |
|
| 158 | -frontend http_in |
|
| 159 | - bind *:8080 |
|
| 160 | - bind *:8443 ssl crt /etc/letsencrypt/haproxy.pem |
|
| 161 | - mode http |
|
| 162 | - |
|
| 163 | - capture request header Host len 256 |
|
| 164 | - capture request header User-Agent len 256 |
|
| 165 | - |
|
| 166 | - acl acme_pth path_beg -i /.well-known/acme-challenge |
|
| 167 | - acl domain1_hdr hdr(host) -i domain1.example.com |
|
| 168 | - acl domain2_hdr hdr(host) -i domain2.example.com |
|
| 169 | - |
|
| 170 | - redirect scheme https code 301 if !{ ssl_fc } !acme_pth |
|
| 171 | - |
|
| 172 | - use_backend letsencrypt if acme_pth |
|
| 173 | - use_backend domain1 if domain1_hdr |
|
| 174 | - use_backend domain2 if domain2_hdr |
|
| 175 | - |
|
| 176 | -backend letsencrypt |
|
| 177 | - server letsencrypt letsencrypt:8000 resolvers docker |
|
| 178 | - |
|
| 179 | -backend domain1 |
|
| 180 | - server domain1-1 container1:5000 resolvers docker check |
|
| 181 | - |
|
| 182 | -backend domain2 |
|
| 183 | - server domain2-1 container2:3000 resolvers docker check |
|
| 184 | -``` |
|
| 185 | - |
|
| 186 | -What's going on here then? |
|
| 187 | -* The global section logs everything to stdout, because that's what you do with docker. [rule 6](/rules#6-thou-shalt-respect-the-sanctity-of-stdout) does not apply in dockerland. |
|
| 188 | -* We're setting the Mozilla recommended ciphers and DH values. Check the [current recommendations](https://mozilla.github.io/server-side-tls/ssl-config-generator/) if you're mental enough to go into production with this stuff. |
|
| 189 | -* We're using 'resolvers' and 'default-server init-addr none' to get around the problem of containers not being up at haproxy startup time. Docker with user-defined networks always puts a resolver at 127.0.0.11:53, and haproxy can use that to resolve container names at *runtime* instead of *startup* time. |
|
| 190 | -* We're *not* running 'check' on the letsencrypt backend - it will be down most of the time, and we don't care. |
|
| 191 | -* We're binding to port 8080 and 8443, and setting the cert to the Let's Encrypt cert we dumped out in the previous section. The ports will be mapped back to 80 and 443 by docker later on. |
|
| 192 | -* Always redirect to https. |
|
| 193 | -* All traffic that matches the certbot [ACME](https://ietf-wg-acme.github.io/acme/draft-ietf-acme-acme.html) challenge protocol is directed to our letsencrypt container (to be created later). |
|
| 194 | -* Other traffic is matched by request hostname to their respective containers. Your routing will probably be more complicated than this, but it's a start. |
|
| 195 | - |
|
| 196 | -Let's wrap this up in docker-compose... |
|
| 197 | - |
|
| 198 | -## docker-compose.yml |
|
| 199 | -In your current directory, put this is `docker-compose.yml`: |
|
| 200 | - |
|
| 201 | -```yaml |
|
| 202 | -version: '3' |
|
| 203 | - |
|
| 204 | -services: |
|
| 205 | - haproxy: |
|
| 206 | - container_name: haproxy |
|
| 207 | - image: haproxy:2.0 |
|
| 208 | - restart: always |
|
| 209 | - volumes: |
|
| 210 | - - ./haproxy/bind:/usr/local/etc/haproxy:ro,Z |
|
| 211 | - - letsencrypt_etc:/etc/letsencrypt:ro |
|
| 212 | - networks: |
|
| 213 | - - haproxy |
|
| 214 | - ports: |
|
| 215 | - - 80:8080 |
|
| 216 | - - 443:8443 |
|
| 217 | - user: '1001' |
|
| 218 | - |
|
| 219 | -volumes: |
|
| 220 | - letsencrypt_etc: |
|
| 221 | - |
|
| 222 | -networks: |
|
| 223 | - haproxy: |
|
| 224 | -``` |
|
| 225 | -Here we have: |
|
| 226 | -* The container_name is 'haproxy'. We'll be referring to this container name later on for sending signals when certs are renewed (referenced in the deploy-hook script from stage 1). |
|
| 227 | -* The 'haproxy/bind' dir is mounted at /usr/local/etc/haproxy, so the haproxy.cfg file we created is in the right place for haproxy to read it. Mounted read-only, and with the 'Z' selinux flag (I'm running RedHat-ish host OSes here, so it's required - leave off the ',Z' if docker complains). |
|
| 228 | -* The letsencrypt volume is mounted at /etc/letsencrypt so haproxy can read the cert file. |
|
| 229 | -* We're creating a user-defined network called 'haproxy' so we can talk to other containers and have built-in dns work. |
|
| 230 | -* The high port numbers are mapped down to the usual 80/443 . |
|
| 231 | -* We're setting a non-priv UID to run as. Because [containers don't need to run as root](https://medium.com/@mccode/processes-in-containers-should-not-run-as-root-2feae3f0df3b). |
|
| 232 | - |
|
| 233 | -## Go! |
|
| 234 | -Run that bad boy with `docker-compose up`. You should see some startup messages and hopefully no errors. haproxy might complain about the backends being down, but that's OK for now. |
|
| 235 | - |
|
| 236 | -Test port 443 from the docker host with: |
|
| 237 | -`openssl s_client -connect localhost:443 | openssl x509 -text` |
|
| 238 | -and you should see your cert if all has gone well. |
|
| 239 | - |
|
| 240 | -Bring haproxy back down with `docker-compose stop` so we've got a clean slate for the next stage. |
|
| 241 | - |
|
| 242 | -# Stage 3 - automatic cert renewal |
|
| 243 | -So far we've got haproxy up, with certs, and everything is just [tickety boo](https://en.wiktionary.org/wiki/tickety-boo). |
|
| 244 | -Those certs only last for 90 days though, and we're not in the habit of breaking [rule 7](/rules#7-thou-shalt-automate-everything). We'll need a container that can: |
|
| 245 | -* See the certificates we already have. |
|
| 246 | -* Renew them. |
|
| 247 | -* Tell haproxy something has changed. |
|
| 248 | -* Keep doing the above. |
|
| 249 | - |
|
| 250 | -## Dockerfile |
|
| 251 | -We've already built the image for this in stage 1, so we're good to go. |
|
| 252 | - |
|
| 253 | -## docker-compose.yml |
|
| 254 | -Add a letsencrypt stanza to the existing `docker-compose.yml`, so it looks like: |
|
| 255 | -```yaml |
|
| 256 | -version: '3' |
|
| 257 | - |
|
| 258 | -services: |
|
| 259 | - haproxy: |
|
| 260 | - container_name: haproxy |
|
| 261 | - image: haproxy:2.0 |
|
| 262 | - restart: always |
|
| 263 | - volumes: |
|
| 264 | - - ./haproxy/bind:/usr/local/etc/haproxy:ro,Z |
|
| 265 | - - letsencrypt_etc:/etc/letsencrypt |
|
| 266 | - networks: |
|
| 267 | - - haproxy |
|
| 268 | - ports: |
|
| 269 | - - 80:8080 |
|
| 270 | - - 443:8443 |
|
| 271 | - user: '1001' |
|
| 272 | - |
|
| 273 | - letsencrypt: |
|
| 274 | - build: ./letsencrypt |
|
| 275 | - image: letsencrypt |
|
| 276 | - container_name: letsencrypt |
|
| 277 | - restart: always |
|
| 278 | - volumes: |
|
| 279 | - - letsencrypt_etc:/etc/letsencrypt |
|
| 280 | - - /var/run/docker.sock:/var/run/docker.sock:rw,Z |
|
| 281 | - networks: |
|
| 282 | - - haproxy |
|
| 283 | - command: bash -c 'sleep 10; while true; do certbot renew --standalone --http-01-port 8000 --deploy-hook /deploy-hook; sleep 86400; done' |
|
| 284 | - privileged: true |
|
| 285 | - |
|
| 286 | -volumes: |
|
| 287 | - letsencrypt_etc: |
|
| 288 | - |
|
| 289 | -networks: |
|
| 290 | - haproxy: |
|
| 291 | -``` |
|
| 292 | -What doing? |
|
| 293 | -* We're mounting the letsencrypt volume back up at /etc/letsencrypt so 'certbot --renew' can operate on the certs. |
|
| 294 | -* The docker socket from the host is mounted at /var/run/docker.sock. This lets us do docker operations from inside the container. |
|
| 295 | -* There's a small sleep to let haproxy start up (ewww, but also, whatever), then we attempt a renew and run the deploy-hook script (see stage 1) if anything changed. |
|
| 296 | -* The deploy-hook script concatenates the cert chain and key into an haproxy style .pem file, then sends a SIGHUP via the docker command to the haproxy container, telling haproxy to re-read its config and pick up the new certs |
|
| 297 | -* The container is granted privileged permissions to let the docker socket work. |
|
| 298 | - |
|
| 299 | -## Go! |
|
| 300 | -Run `docker-compose up` to bring up haproxy and the letsencrypt container. certbot will (after 10 seconds) read the current certs and decide there's nothing to do, then go to sleep for a day. |
|
| 301 | -haproxy should start up and tell you nice things about the letsencrypt backend being available. |
|
| 302 | - |
|
| 303 | -# Add new domains and containers |
|
| 304 | -You're on your own with the container, but haproxy and certbot config we can do. |
|
| 305 | - |
|
| 306 | -## DNS |
|
| 307 | -Make sure the new domain's A record is pointing at haproxy's IP. |
|
| 308 | - |
|
| 309 | -## haproxy |
|
| 310 | -Add the new domain to haproxy.conf: |
|
| 311 | -```text |
|
| 312 | -... |
|
| 313 | - acl logs_hdr hdr(host) -i new.domain.com |
|
| 314 | -... |
|
| 315 | - use_backend new-container if logs_hdr |
|
| 316 | -... |
|
| 317 | -backend new-container |
|
| 318 | - server new-container1 new-container:6666 resolvers docker check |
|
| 319 | -``` |
|
| 320 | - |
|
| 321 | -## Container |
|
| 322 | -Bring up the new container if it's not already running. |
|
| 323 | - |
|
| 324 | -## letsencrypt |
|
| 325 | -Log into the letsencrypt container: |
|
| 326 | -`docker exec -ti letsencrypt bash` |
|
| 327 | - |
|
| 328 | -Then run certbot with the new domain and run the hook script: |
|
| 329 | -```bash |
|
| 330 | -$ certbot --standalone --expand \ |
|
| 331 | --d "domain1.example.com" \ |
|
| 332 | --d "domain2.example.com" \ |
|
| 333 | --d "new.domain.com" |
|
| 334 | -$ /deploy-hook |
|
| 335 | -``` |
|
| 336 | - |
|
| 337 | -# Next? |
|
| 338 | -Do what you like. I'm not the boss of you. |
|
| 339 | - |
|
| 340 | -# Relax |
|
| 341 | - |