A Deploy Script Was Silently Rolling Back Our Let's Encrypt Renewals

A staging smoke test failed TLS verification. The certificate had expired five weeks earlier, even though Caddy had renewed it successfully since then. The deploy script itself deleted each renewal on the next run.

The restore-on-deploy snapshot loop

The service is a single Amazon EC2 instance behind Caddy with automatic HTTPS: an Elastic IP, an Amazon Route 53 A record, and a security group open on ports 80 and 443. It holds long-lived WebSocket sync connections, and an Application Load Balancer would cost about as much as the instance itself, so TLS terminates directly on the box.

Deploys run over AWS Systems Manager (SSM): docker-compose down, then up. To survive instance replacement, the deploy script included a disaster-recovery step around the Caddy data volume:

# deploy script, every run:
1. rm -rf the caddy data volume contents
2. untar the last S3 snapshot into it          <- restore
3. docker-compose up
4. tar the volume, upload to S3                <- backup

The timeline from the certificates showed the issue clearly: issued April 29, expiring July 28. The expired certificate served to clients had notBefore Apr 29 17:24:38, the day the snapshot was first taken. Every deploy after that restored April, ran for minutes, and backed up April again.

The design could never converge. Caddy renews certificates on a 30-day cycle into that volume. But the backup ran minutes after the restore, inside the same deploy. A renewal only survives into the snapshot if it happens between step 3 and step 4 of a single deploy: a window of minutes, against a renewal cadence of weeks. The snapshot was a fixed point.

renewal (July) -> volume -------+
                                |  next deploy:
process memory ->+              +-- rm -rf -> untar April -> up -> tar April -> S3
                 |              |
S3 snapshot -----+--- April ----+      (renewal gone before it was ever backed up)

Caddy did its job. Renewal succeeded, issuance succeeded after expiry, and the running process held fresh material. None of it could outlive the next deploy.

Two compounding blind spots:

• The ACME account was anonymous: the Caddyfile never configured email in its entire git history, so Let’s Encrypt had no contact for expiry notices.
• The only detector was the deploy’s own smoke test, which validates TLS on deploy. An expired certificate sits silently between deploys.

One more trap during recovery: after a forced re-issuance, a retry can fail with Let’s Encrypt’s duplicate certificate rate limit (5 per week). A rerun that “suddenly passes” usually means the rate-limit window cleared rather than the configuration changing.

Gating volume restore and planning stateless storage

Phase 1: stop the rollback. The restore became a seed: it only runs when the volume has no Caddy state directory at all. A read-only probe (test -d via a throwaway container) checks that, and a failed probe fails the deploy loudly: an unreadable volume should never be wiped on a guess. The Caddyfile also registers an ACME email now.

Phase 2: decouple storage ownership. The durable fix is that the deploy script should not own certificate state at all. Specced for follow-up:

• Caddy’s storage interface backed by S3 via a community plugin. The bucket becomes the single source of truth, the container becomes stateless, and the backup/restore step disappears because the storage is the database.
• DNS-01 challenges via Route 53. Issuance no longer depends on inbound ports, a healthy service, or a warm volume.

Verifying the plan surfaced several operational constraints:

• The module name I remembered (caddy-storage-s3) does not exist. The caddy.storage.s3 namespace has three community forks, and exactly one can be linked per build.
• The techknowlogick/certmagic-s3 fork supports instance-profile authentication because all credential configuration is optional and the plugin uses the aws-sdk-go-v2 default chain. Confirmed from the go.mod dependency tree rather than documentation claims.
• The IAM delta is exactly one action (s3:DeleteObject), computable from the plugin’s documented S3 operation list.

AWS Certificate Manager (ACM) private keys are non-exportable, so ACM works only on AWS-integrated services. Using ACM requires an Application Load Balancer in front of the box, reintroducing the monthly cost the direct-termination design avoided.

How poisoned state wedged CertMagic at boot

The merged fix refused to wipe the volume, exactly as designed. The first deploy running it failed anyway, because the state it now protected was poisoned. On the instance, docker logs told the rest:

# at boot, caddy's storage maintenance saw the restored April cert:
"certificate expired beyond grace period; cleaning up"
"deleting asset because resource expired"     # .crt, .key, .json: gone
# then, every retry, for up to 30 days:
"error": "open /data/caddy/certificates/acme.zerossl.com-v2-dv90/...key:
         no such file or directory"

Three facts chain together:

• CertMagic deletes certificates expired beyond its 14-day grace period at boot.¹ It removed the April certificate files from disk seconds before the deploy’s backup probe scanned the volume, which is why the probe reported no certificates.
• Renewal then wedged in a local loop: it tried to read a key from a CA directory that never existed, failed with ENOENT, and backed off exponentially. It never reached the network.
• The process kept serving the deleted, expired certificate from its in-memory cache. Expired on the wire, empty on disk, wedged in between.

Recovery was the part I least expected: wipe the data volume and restart the container. Empty storage is a valid state for Caddy, which obtains a fresh certificate from scratch. Six seconds later it had one. The old destructive restore had been accidentally working around this CertMagic failure mode: a wiped volume cannot wedge on stale state.

One forensic trap worth recording: Let’s Encrypt backdates notBefore by an hour.² I read a fresh certificate’s notBefore as wall-clock time and misplaced my own fix by an hour in the timeline. When ordering events, trust file mtimes and process logs, not certificate dates.

Verification in deploy logs

Verified in the deploy logs after the recovery run:

• The probe logs restore.skipped_volume_present; the deploy no longer touches certificate state. A renewal survives every deploy by construction.
• The backup step found the fresh cert on its first attempt and uploaded it: the S3 snapshot now holds current state, so instance replacement is safe again.
• The smoke test passed on the first attempt, and the new ACME account carries the configured email, which only takes effect at account creation.

Operational takeaways

• Treat “restore runs unconditionally” as a design smell. A backup is write-often, read-rarely. If restore sits on the hot path, the snapshot is a write target being chased by a rolling snapshot of itself: a fixed point, not a backup.
• Add a --min-days-valid pre-check to the smoke test. The TLS check caught rot at expiry; 21 days of margin turns that into an early warning, no email required.
• An expired certificate cannot be safely preserved. Beyond CertMagic’s grace period it gets deleted at boot, and renewal can wedge after the deletion. If a box ever serves an expired certificate, wipe its certificate storage and let it re-issue: waiting cannot unwedge it.
• Verify a plugin exists before planning around it. One curl to the Go proxy corrected the module name, surfaced a three-way fork choice, and converted two “should work” assumptions into verified facts.

References

• Caddy Automatic HTTPS. The renewal model the deploy was fighting.
• Caddy storage conventions. The storage interface Phase 2 replaces with S3.
• caddy.storage.s3 module docs. Namespace shared by three forks.
• techknowlogick/certmagic-s3. The fork selected for optional credentials and aws-sdk-go-v2 support.
• caddy-dns/route53. DNS-01 challenge plugin for Route 53.
• Let’s Encrypt rate limits. The duplicate-certificate limit that wedged the first recovery retry.
• CertMagic. Caddy’s certificate library, whose grace-period cleanup and renewal retry backoff produced the wedge.
• Mozilla Bugzilla #1715455. Let’s Encrypt incident disclosure documenting 1-hour backdating of notBefore.
• Amazon EC2. Host compute platform.
• Amazon Route 53. DNS management.
• AWS Systems Manager. Deployment mechanism via SSM.
• AWS ACM concepts. Non-exportable keys and integrated services constraint.