Troubleshooting

This page maps common Meridian failure messages to the fastest diagnostic path. Examples use my-app, prod-01.example.com, and my-app.example.com; replace them with your service, host, and domain.

Healthcheck Timeout

Problem: deploy fails with Health check failed for my-app-green: needed 3 consecutive successes.

Root cause: the new app container did not return enough consecutive successful responses from servers.web.proxy.healthcheck.path before the rollout timeout.

Diagnose:

meridian logs --host prod-01.example.com
ssh deploy@prod-01.example.com 'systemctl --user status my-app-green.service'
ssh deploy@prod-01.example.com 'journalctl --user -u my-app-green.service -n 100 --no-pager'
ssh deploy@prod-01.example.com 'podman run --rm --network=meridian-proxy docker.io/library/alpine:3.21 wget -S -O- http://my-app-green:8000/health'

Fix:

# Fix the app so the health route returns 200 without depending on slow startup work.
curl -i http://localhost:8000/health

# If the route or port is different, update .meridian/deploy.yml:
# servers.web.proxy.app_port and servers.web.proxy.healthcheck.path
meridian check
meridian deploy

See the deploy.yml healthcheck reference for field details.

Recreate Deploy Left The Route In Maintenance

Problem: a strategy: recreate deploy reports that maintenance began and the service remains in maintenance.

This is deliberate. Once Meridian has stopped old app processes, hooks or the new container may have migrated the database or persistent volumes. Meridian therefore does not restart the old image, perform an image-only rollback, or run kamal-proxy resume automatically.

Diagnose every app role named in the deploy error:

ssh deploy@prod-01.example.com \
  'systemctl --user status my-app-blue.service my-app-green.service my-app-cron.service'
ssh deploy@prod-01.example.com \
  'journalctl --user -u my-app-blue.service -u my-app-green.service -u my-app-cron.service -n 100 --no-pager'
meridian audit --host prod-01.example.com --lines 50

Repair and verify the new release in place. If data restoration is required, restore the matching image, database, and persistent volumes from the same backup. Only after the service is healthy should you reopen traffic:

ssh deploy@prod-01.example.com \
  'podman exec kamal-proxy kamal-proxy resume my-app'

Do not resume merely to clear the maintenance response: it exposes whichever target is currently registered. Recreate is single-host-only in this release, and Accessories remain active while the app route is stopped.

manifest-collisions: fail

Problem: meridian check reports manifest-collisions: fail for a host that already runs Meridian services.

Root cause: Meridian found another service manifest claiming the same proxy host/path, or found a stale manifest written by an older deploy shape.

Diagnose:

meridian check
ssh deploy@prod-01.example.com 'find ~/.local/state/meridian/services -name manifest.json -maxdepth 3 -print'
ssh deploy@prod-01.example.com 'cat ~/.local/state/meridian/services/my-app/manifest.json'

Fix:

# If another live service owns the same proxy host/path, change one deploy.yml first.
meridian plan

# If this is only a stale manifest for the affected service, redeploy that service.
meridian deploy
meridian check

Per-service manifests live under ~/.local/state/meridian/services/<service>/manifest.json and are refreshed on the next successful deploy of that service.

image not known During Stream Or Incremental Transfer

Problem: deploy fails mid-transfer with image not known, or meridian check reports a missing local image.

Root cause: transfer.mode: stream and transfer.mode: incremental read from local Podman image storage; Meridian cannot stream an image that only exists in a registry or Docker storage.

Diagnose:

meridian plan
podman image exists ghcr.io/acme/my-app:latest
meridian check

Fix:

podman build -t ghcr.io/acme/my-app:latest .
podman image exists ghcr.io/acme/my-app:latest
meridian check
meridian deploy

If you want hosts to pull from a registry instead, remove transfer.mode or set it to registry; see the transfer reference.

Hostname Lookup ... Try Again In App Logs

Problem: app logs contain messages like Hostname lookup for postgres failed: Try again right after a deploy or restart.

Root cause: rootless Podman's aardvark-dns can briefly lag behind container startup, especially when the app starts before its co-network accessories are ready.

Diagnose:

meridian logs --host prod-01.example.com
meridian accessory logs postgres
ssh deploy@prod-01.example.com 'podman ps --format "{{.Names}}\t{{.Networks}}"'
ssh deploy@prod-01.example.com 'podman network inspect my-app'

Fix:

# Start the accessory before deploying the app.
meridian accessory start postgres

# Declare readiness for accessories on my-app.network, then verify before deploy.
meridian plan
meridian check
meridian deploy

The accessory readiness gate waits before starting the new app color; see Accessory readiness.

kamal-proxy bind: permission denied On Port 80

Problem: meridian setup or kamal-proxy startup fails with bind: permission denied for :80.

Root cause: rootless containers need low-port binding enabled on the server before kamal-proxy can listen on ports 80 and 443.

Diagnose:

ssh deploy@prod-01.example.com 'systemctl --user status kamal-proxy.service'
ssh deploy@prod-01.example.com 'journalctl --user -u kamal-proxy.service -n 100 --no-pager'
ssh deploy@prod-01.example.com 'sysctl net.ipv4.ip_unprivileged_port_start'

Fix:

meridian server bootstrap --host prod-01.example.com
meridian setup
meridian check

Bootstrap is what sets net.ipv4.ip_unprivileged_port_start; re-running it on an already-provisioned host is safe. setup then writes and starts the proxy Quadlet.

Lets Encrypt Issuance Hangs

Problem: deploy reaches the proxy switch but HTTPS certificate issuance appears to hang or fail.

Root cause: proxy.host and, when configured, assets.host must already resolve to the server before kamal-proxy asks Lets Encrypt for certificates.

Diagnose:

dig +short my-app.example.com
dig +short assets.my-app.example.com
ssh deploy@prod-01.example.com 'curl -I http://my-app.example.com'
ssh deploy@prod-01.example.com 'journalctl --user -u kamal-proxy.service -n 100 --no-pager'

Fix:

# In your DNS provider, point my-app.example.com and assets.my-app.example.com at prod-01.
dig +short my-app.example.com
dig +short assets.my-app.example.com
meridian deploy

Do not enable ssl: true until public DNS points at the target host.

Distroless Or Scratch Image Has No curl Or wget

Not actually a problem, but it comes up often enough to answer here: Meridian never runs curl or wget inside your app image. The health probe runs from a temporary container on meridian-proxy and reaches the app over the network, so a distroless or FROM scratch image with no shell works fine.

The one case where it does break is when the probe image cannot be pulled on the host.

Diagnose:

meridian plan
ssh deploy@prod-01.example.com 'podman image exists docker.io/library/alpine:3.21'
ssh deploy@prod-01.example.com 'podman run --rm --network=meridian-proxy docker.io/library/alpine:3.21 wget -S -O- http://my-app-green:8000/health'

Fix:

# Do not add shell tools to the app image just for Meridian.
# If the default probe image is unavailable on your hosts, mirror or override it.
podman pull docker.io/library/alpine:3.21
meridian check
meridian deploy

The relevant field is servers.<role>.proxy.healthcheck.probe_image; see Health check tuning.

Stale Deploy Lock

Problem: deploy exits because another deploy lock is held, but no deploy appears to be running.

Root cause: a previous deploy was interrupted after acquiring the remote lock and before releasing it.

Diagnose:

meridian lock status
meridian audit --host prod-01.example.com --lines 50
ssh deploy@prod-01.example.com 'ps -fu "$USER" | grep meridian'

Fix:

# Only release the lock after confirming no deploy/rollback/proxy mutation is still running.
meridian lock release
meridian check
meridian deploy

If you are not sure whether a mutation is still active, inspect the audit log and systemd state before releasing the lock.

CSS url() Assets 404 Against The Asset CDN

Problem: after enabling assets:, the page's HTML loads, but images or fonts referenced from inside a CSS file (url("../images/logo.png")) return 404 from the asset host.

First verify the Marten production settings. The app image must contain the manifest generated by collectassets --fingerprint, and Marten must use the same hostname as Meridian's assets.host:

config.assets.url = "https://assets.my-app.example.com/"
config.assets.manifests = ["src/manifest.json"]

If HTML template assets already resolve to that host with fingerprinted names, the remaining root cause is CSS processing. Marten's {% asset %} tag runs in HTML templates, but it does not rewrite references inside .css files. A raw url("../images/logo.png") therefore keeps its un-fingerprinted name, which is not present in the published asset release.

Diagnose:

# the rendered HTML points at the asset host and fingerprinted names…
curl -s https://my-app.example.com | grep -o 'https://assets\.my-app\.example\.com/[^"]*'
# …but the CSS still references the raw, un-fingerprinted path
curl -s https://assets.my-app.example.com/app-<hash>.css | grep -o 'url([^)]*)'

Fix: don't reference fingerprinted assets from inside CSS files. Resolve the URL in the HTML layer, where the framework can fingerprint it, and hand it to CSS via a custom property set from a <style> block in base.html:

<!-- base.html -->
<style>
  :root {
    --logo-url: url("{% asset "images/logo.png" %}");
  }
</style>

.brand { background-image: var(--logo-url); }

The path is resolved through Marten's manifest in the template; the CSS just consumes the resulting URL.

Asset CDN Still Serves Old Headers After Meridian Update

Problem: after updating Meridian (for example to pick up a Caddyfile template change), the asset CDN still returns the old headers — no Content-Encoding: zstd after the compression patch, or no Access-Control-Allow-Origin: * after the CORS patch — even though meridian deploy reported success.

Root cause: meridian deploy regenerates the Caddyfile on the server and writes it to its mount path (~/.config/containers/<service>-assets-caddy/Caddyfile). The <service>-assets-server container is already running; Caddy reads its configuration only at process start, not when the underlying file changes. Nothing triggers a re-read automatically.

Diagnose:

# 1. Confirm the new Caddyfile content is on the server:
ssh deploy@prod-01.example.com \
  'cat .config/containers/my-app-assets-caddy/Caddyfile'

# 2. Confirm the running asset-server still serves old headers:
curl -sI -H "Accept-Encoding: zstd, gzip" \
  https://assets.my-app.example.com/css/app.<hash>.css \
  | grep -iE "content-encoding|cache-control|access-control"

If the on-disk Caddyfile shows the new directives but the curl response does not, the asset-server has not picked up the new config yet.

Fix:

ssh deploy@prod-01.example.com \
  'systemctl --user restart my-app-assets-server.service'

Caddy starts up against the new Caddyfile (~500 ms of asset-server downtime; cached browser responses keep working through the gap because the asset CDN sets Cache-Control: public, max-age=31536000, immutable).

This restart is not needed on a normal meridian deploy — only when the Caddyfile content itself has changed. That happens at most a couple of times per year, typically when picking up a Meridian release that touches src/quadlet/templates/assets_caddy_config_file.ecr.