Rails, Kamal & TLS

I have a really very trivial Rails application that I want to deploy as a container with Kamal. This is fairly painless to do by following along with the demo, but DHH glosses over TLS setup by using a domain hosted in CloudFlare to provide TLS.

I don't register my domains in CloudFlare – I need another way. Here I go through the steps to setup my application and configure Let's Encrypt. I am indebted to this blog post by Guillaume Briday which helped me enormously. This post is a re-telling of their setup as I walked through it.

Install Kamal and initialise

My Rails app should define its dependencies so the first thing I'm going to do is add Kamal to my Gemfile and pin it to version. From here on out, I'm going to prefix all my Kamal commands with bundle exec to make sure I know I using the right version.

bundle add kamal

Now I'm going to get Kamal to create its default deployment script

bundle exec kamal init

Which will generate a file that looks like this after a bit of editing (commented lines removed):

service: dna-tools

image: nrbrookes/dna-tools

servers:
  - 65.100.100.111
registry:
  username: nrbrookes
  password:
    - KAMAL_REGISTRY_PASSWORD

env:
  secret:
    - RAILS_MASTER_KEY

volumes:
  - storage:/rails/storage

I'm using Docker Hub as my registry, the default, and I've generated a Docker Hub application password which I'm setting in my .env as the KAMAL_REGISTRY_PASSWORD.

My RAILS_MASTER_KEY is also set in the .env.

I've also got Docker running locally and, as I'm deploying on different architecture, Kamal will automatically use a build kit to create the image.

One edit I have done is to set a volume. I'm not using Active Record right now, and if I do, it'll just be SQLite, so I'm setting a volume to where my database is to persist it across deployments.

I have my SSH keys setup and ready to go on my cheap-as-chips Hetzner machine, so I'll I need to do to prepare the server is:

bundle exec kamal server bootstrap

Tweak Rails configuration

In this case, I want Traefik handle the TLS with Let's Encrypt. For the moment, to get things up and running, I want to run rails without the Force TLS (SSL) option on. I'm going to disable it now and will reenable it later.

# config.force_ssl = true

Trying it out

At this point, I'm going to run:

bundle exec kamal setup

This deploys everything to the server and brings up Traefik and the application. I should be able to connect to this at port 80 like a plain HTTP website and see the root page.

One change I had to do was to increase the SSL timeout on my Debian server which aggressively closed connections after a few seconds and Kamal won't reconnect.

Configure Traefik for TLS

So now I want to turn on TLS/SSL using Traefik to handle all the Let's Encrypt stuff.

Traefik router rules

First thing to do is to rearrange my servers block in the deploy.yml. This Kamal document tells me "Traefik will only by default be installed and run on the servers in the web role (and on all servers if no roles are defined)." So I might want to run more servers in the future (database, redis server for e.g.) so I'll change this section:

servers:
  web:
    hosts:
      - 65.100.100.111

This Kamal document tells me I can use labels to control Traefik's router rules. Let's add some to:

1. Create an entry point called websecure (more on this later).
2. Create a rule that matches incoming requests (the hostname in this case) which we'll need for Let's Encrypt to run the certificate generation.
3. Set Let's Encrypt as the certificate resolver.

This is what the servers block now looks like:

servers:
  web:
    hosts:
      - 65.100.100.111
    labels:
      traefik.http.routers.dna.entrypoints: websecure
      traefik.http.routers.dna.rule: Host(`dna.billy-ruffian.co.uk`)
      traefik.http.routers.dna.tls.certresolver: letsencrypt

Traefik arguments

Kamal builds Traefik with a sensible set of defaults. We need to change them and add some arguments to configure the behaviour:

traefik:
  options:
    publish:
      - "443:443"
    volume:
      - "/letsencrypt/acme.json:/letsencrypt/acme.json" 
  args:
    entryPoints.web.address: ":80"
    entryPoints.websecure.address: ":443"
    entryPoints.web.http.redirections.entryPoint.to: websecure 
    entryPoints.web.http.redirections.entryPoint.scheme: https
    entryPoints.web.http.redirections.entrypoint.permanent: true
    certificatesResolvers.letsencrypt.acme.email: "bellerophon@billy-ruffian.co.uk"
    certificatesResolvers.letsencrypt.acme.storage: "/letsencrypt/acme.json"
    certificatesResolvers.letsencrypt.acme.httpchallenge: true
    certificatesResolvers.letsencrypt.acme.httpchallenge.entrypoint: web

The options section contains Docker arguments. Here, I'm binding port 443 of the host to port 443 of the Traefik container. I'm also setting up another volume to store the Let's Encrypt certificates.

The args section is the part that configures Traefik. Firstly, the web and websecure ports are set to 80 and 443. Then traffic on the web (port 80) is set to permanently redirect to websecure (port 443) with the https scheme.

The acme (Let's Encrypt) resolver is configured. I tell it my email address and to store the certificates in the volume I created earlier. I configured the HTTP Challenge (a challenge must exist, documentation here). Importantly, the certificatesResolvers.letsencrypt.acme.httpchallenge.entrypoint value must match the role in the servers block which we explicitly defined as web.

Create a directory on the VPS for Let's Encrypt

I needed to SSH to the VPS to create the physical directory for the certificates to live in (as root or with sudo):

mkdir -p /letsencrypt && touch /letsencrypt/acme.json && chmod 600 /letsencrypt/acme.json

Reboot Traefik

One advantage of Traefik is that an application deployment is seamless – Traefik isn't rebooted when newer versions of the application are deployed and the new version of the application is booted before the old one taken down. Since the Traefik configuration has changed, it needs a manual reboot:

bundle exec kamal traefik reboot

Now do a proper deploy:

bundle exec kamal deploy

It took a moment or two following my first request to set up the initial certificate. After that, it was swinging with HTTPS. Traefik will auto renew the certificate going forward.

Untweak Rails configuration

Don't miss this step. We can now tell Rails to only accept connections over TLS. Reenabling this configuration changes some important things around cookie handling (setting the secure flag). We need to uncomment this line:

config.force_ssl = true

Now do another deploy:

bundle exec kamal deploy

Wrapping up

Here are the steps I needed to follow:

1. Turn off Force TLS in the production Rails environment
2. Get a the application deploying successfully with only HTTP
3. Add the Let's Encrypt configuration to Traefik in deploy.yml
4. Create a directory on the server to store the certificates
5. Reboot Traefik and deploy the application and give it a test
6. Reenable TLS in Rails and redeploy.

My final deploy.yml:

service: dna-tools
image: nrbrookes/dna-tools

servers:
  web:
    hosts:
      - 65.100.100.111
    labels:
      traefik.http.routers.dna.entrypoints: websecure
      traefik.http.routers.dna.rule: Host(`dna.billy-ruffian.co.uk`)
      traefik.http.routers.dna.tls.certresolver: letsencrypt


registry:
  username: nrbrookes
  password:
    - KAMAL_REGISTRY_PASSWORD

env:
  secret:
    - RAILS_MASTER_KEY

volumes:
  - storage:/rails/storage

traefik:
  options:
    publish:
      - "443:443"
    volume:
      - "/letsencrypt/acme.json:/letsencrypt/acme.json" 
  args:
    entryPoints.web.address: ":80"
    entryPoints.websecure.address: ":443"
    entryPoints.web.http.redirections.entryPoint.to: websecure 
    entryPoints.web.http.redirections.entryPoint.scheme: https
    entryPoints.web.http.redirections.entrypoint.permanent: true
    certificatesResolvers.letsencrypt.acme.email: "bellerophon@billy-ruffian.co.uk"
    certificatesResolvers.letsencrypt.acme.storage: "/letsencrypt/acme.json" 
    certificatesResolvers.letsencrypt.acme.httpchallenge: true
    certificatesResolvers.letsencrypt.acme.httpchallenge.entrypoint: web