One VPS, rootless Docker and Caddy: how my server is set up

I run everything on one VPS. This site, a couple of other apps, a database, a cache, and the reverse proxy in front of all of it. It is an 8 vCPU box with 23 GB of RAM on Ubuntu 26.04, with Cloudflare in front. Right now it is using under 3 GB of that memory.

I know the arguments for spreading things out. For my workload they don’t win. One machine means one place to look when something breaks, one firewall, one set of logs, and one bill. The flip side is that if the box dies, everything is down at once. Rebuilding it means a few compose files, one Caddyfile and the data volumes. I can live with that.

This post is the actual setup, not a template. App names are swapped for myapp, but the config is what runs.

Rootless Docker

The Docker daemon runs as my normal user, not as root. The system-wide docker.service and docker.socket are both disabled, and my user is not in the docker group. The only daemon on the box is the rootless one, started by systemd as a user service:

$ systemctl --user status docker
● docker.service - Docker Application Container Engine (Rootless)
     Loaded: loaded (~/.config/systemd/user/docker.service; enabled)
     Active: active (running)

The reason is simple. With classic Docker, anyone who can talk to the socket is effectively root on the host. Mounting / into a container is one command. With rootless mode, a container escape lands you as an unprivileged user who can’t touch system files. I’d rather not find out how good my images are at staying contained.

Two things make it work across reboots and logins. First, lingering has to be on, otherwise systemd kills your user services when you log out and doesn’t start them at boot:

$ loginctl enable-linger $USER

Second, the CLI has to know where the socket is. I have this in .bashrc, and the Docker context is set to rootless as well, so either way commands go to the right daemon:

export DOCKER_HOST=unix:///run/user/$(id -u)/docker.sock

Ports 80 and 443 without root

This is the first thing that trips everyone up. An unprivileged process can’t bind ports below 1024, so out of the box a rootless Caddy can’t listen on 80 or 443.

There are a few ways around it. You can give rootlesskit the CAP_NET_BIND_SERVICE capability, or run Caddy on high ports and redirect with nftables. I went with the boring one and lowered the threshold with a sysctl:

# /etc/sysctl.d/99-unpriv-ports.conf
net.ipv4.ip_unprivileged_port_start=80

The trade-off is that any local user can now bind 80 and up. On a box where I’m the only human user, that’s fine. On a shared machine I’d think twice.

Rootless networking goes through slirp4netns with the builtin port driver. One side effect I actually like: the published ports are real listening sockets owned by rootlesskit on the host, so ufw’s normal input rules apply to them. With rootful Docker, published ports are wired in through Docker’s own iptables chains and can slip past ufw, which has surprised a lot of people. The known downside of the builtin port driver is that containers don’t see the real client IP. Since Cloudflare is in front and passes the client address in a header, I haven’t cared.

One proxy network, one Caddy

Every container that should be reachable from the internet joins one user-defined bridge network called proxy. Caddy owns that network and is the only container that publishes ports to the host. Its compose file is small:

services:
  caddy:
    image: caddy:2
    container_name: caddy
    restart: unless-stopped
    mem_limit: 256m
    ports:
      - "80:80"
      - "443:443"
      - "443:443/udp"
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile:ro
      - caddy_data:/data
      - caddy_config:/config
    networks:
      - proxy
volumes:
  caddy_data:
  caddy_config:
networks:
  proxy:
    name: proxy

I have nothing against Nginx. It still runs inside the container that serves this site. But as the front door, Caddy wins on one point: certificates are not my problem anymore. You put a hostname in the Caddyfile, reload, and it gets and renews the certificate. No certbot, no cron job, no renewal hook that silently stopped working three months ago. The caddy_data volume keeps the certificates across container rebuilds, which matters, because you don’t want to hit Let’s Encrypt rate limits because you recreated a container a few times.

The other point is that the config stays short. Adding a site is two blocks:

doganaydin.com {
	reverse_proxy myapp:8080
}

www.doganaydin.com {
	redir https://doganaydin.com{uri} permanent
}

Because everything shares the proxy network, Caddy reaches containers by name through Docker’s internal DNS. No IPs, no host ports. There is also a catch-all :80 block that just answers caddy ok, which is handy when I want to check that the proxy itself is alive without involving any app.

It’s one file for all sites. Before larger edits I copy it to a .bak file next to it. Crude, but it means a broken edit is one cp away from undone. Some of the per-app blocks have grown a bit (redirect rules, rewriting a Location header from an app that insisted on leaking its internal address), but even the worst one is about twenty lines.

What an app looks like

Each app lives in its own directory under ~/apps/ with a Dockerfile and a compose file. This is the one for this site, more or less verbatim:

services:
  myapp:
    build: .
    image: myapp:latest
    container_name: myapp
    restart: unless-stopped
    expose:
      - "8080"
    mem_limit: 512m
    read_only: true
    tmpfs:
      - /tmp
    security_opt:
      - no-new-privileges:true
    cap_drop:
      - ALL
    healthcheck:
      test: ["CMD", "wget", "-q", "-O", "/dev/null", "http://127.0.0.1:8080/healthz"]
      interval: 30s
      timeout: 3s
      retries: 3
      start_period: 5s
    networks:
      - proxy

networks:
  proxy:
    external: true

There is no ports: section. expose is documentation more than anything; the app is only reachable from other containers on proxy, which in practice means Caddy. The network is declared external: true so that docker compose down in an app directory never tries to remove the shared network.

Every app gets a mem_limit. Without one, a leaking process can eat the whole box and take the database with it. A container that hits its limit gets killed and restarted, which is a much better failure than the kernel picking a victim for me.

The rest is cheap hardening that costs nothing for a static site: read-only root filesystem, a tmpfs for /tmp, all capabilities dropped, and no-new-privileges. The image itself is a multi-stage build. Node builds the static files, and the runtime stage is nginx-unprivileged running as a non-root user on port 8080. Inside a rootless daemon that’s two layers of “not root”, which is the point.

The healthcheck hits a /healthz location that just returns ok. It’s not deep, but docker ps showing (healthy) next to every app is a nice thing to glance at.

One MariaDB, one Redis

I don’t run a database per app. There’s one MariaDB and one Redis in ~/data/, also on the proxy network, and each project gets its own database inside it. That’s the separation I need. Running five MariaDB instances would mean five buffer pools fighting for memory and five things to back up.

services:
  mariadb:
    image: mariadb:lts
    restart: unless-stopped
    command: ["--character-set-server=utf8mb4",
              "--collation-server=utf8mb4_unicode_ci",
              "--innodb-buffer-pool-size=1G"]
    mem_limit: 2g
    networks: [proxy]
  redis:
    image: redis:7-alpine
    restart: unless-stopped
    command: ["redis-server", "--requirepass", "${REDIS_PASSWORD}",
              "--appendonly", "yes",
              "--maxmemory", "512mb", "--maxmemory-policy", "allkeys-lru"]
    mem_limit: 768m
    networks: [proxy]

Neither publishes a port. To get a SQL shell I docker exec into the container. Passwords live in a .env file next to the compose file with 600 permissions.

Note the gap between the limits and the app settings. MariaDB gets a 1 GB buffer pool and a 2 GB container limit, because it needs memory beyond the buffer pool for connections and sorting. Redis gets maxmemory 512mb under a 768 MB limit, and an LRU policy so it evicts keys instead of running into the wall. If you set those two numbers equal, the container limit wins and you get an OOM kill instead of graceful eviction.

Logs and disk

Docker’s default json-file logging has no size limit, and one chatty container is enough to fill a disk. So the daemon config caps it for every container:

{"log-driver":"local","log-opts":{"max-size":"10m","max-file":"3"}}

For rootless Docker this file lives in ~/.config/docker/daemon.json, not /etc/docker/. Edit the wrong one and your settings are silently ignored.

The thing that actually eats disk is the build cache. Since every deploy builds on the server, old layers pile up quietly. When I last looked, the build cache was about 25 GB, most of it reclaimable, and much bigger than all images and volumes combined. docker builder prune is something I have to remember to run.

Kernel and limits

A few sysctls in their own file:

# /etc/sysctl.d/98-perf.conf
net.core.default_qdisc=fq
net.ipv4.tcp_congestion_control=bbr
fs.nr_open=2097152
fs.file-max=2097152
fs.inotify.max_user_watches=524288
fs.inotify.max_user_instances=1024

BBR with fq is the usual choice for a server talking to the internet. I don’t have benchmarks to show you, it’s just a sensible default that Ubuntu doesn’t turn on for you. The file and inotify limits are there because Node tooling and file watchers run out of inotify watches surprisingly fast, and the error you get doesn’t tell you that’s the problem. The per-process nofile limit is raised to 1048576 in /etc/security/limits.d/ to match.

No swap

There’s no swap on this box. That’s on purpose. Every container has a memory limit, the database and cache are sized well under the total, and there’s plenty of headroom. When something misbehaves I want it to hit its limit and get restarted, not slowly push the whole machine into swap where everything gets slow and nothing actually fails. A dead container shows up right away. A server that’s swapping just feels sluggish and you spend an hour finding out why.

If I ran this on a 2 GB box I’d decide differently. With this much RAM, swap would mostly hide mistakes.

The boring security bits

The firewall is ufw with default deny incoming and only 22, 80 and 443 open. SSH is key-only with root login disabled, set in a small drop-in under /etc/ssh/sshd_config.d/:

PermitRootLogin no
PasswordAuthentication no
KbdInteractiveAuthentication no

fail2ban runs one jail, sshd, reading from the systemd journal and banning through nftables after five failures for an hour. With password auth off, it’s mostly there to keep the logs quiet. Unattended upgrades are on for security patches.

Deploys

Deploys are git pull and docker compose up -d --build in the app directory. The repo is cloned with its own deploy key, so the server’s access is scoped to that repo and not to my whole account. No registry, no CI pushing images, no webhook. For a handful of apps that I deploy when I feel like it, that’s the right amount of machinery. The cost is that builds happen on the production box, which is where that 25 GB of build cache comes from.

If I needed zero-downtime deploys or more than one person shipping, I’d change this. For now, a short blip while a container is recreated is a price I’m happy to pay for having nothing else to maintain.