Immutable release bundles and SSM deploys on EC2 without a container orchestrator

The previous deploy on this single EC2 host was a git pull against a long-lived checkout, with per-database “latest” restores that could mix state across runs. This is the immutable-release and coherent-backup model I put in to make deploys reversible and restores internally consistent.

Deployment constraints

Three things were wrong with the previous day-2 model on a single EC2 host.

Long-lived git checkouts drift. The host ran a git pull-style working copy. Over time the on-disk tree diverged from what CI had actually shipped: uncommitted local edits, a partial pull after a force-push, a stray branch checked out for debugging. “What version is on this host?” became a forensic question, and rollback meant re-running git checkout <prev> against a tree that might not match the previous release.

Per-component “latest backup” restores can mismatch state. The host runs two stateful stores: PostgreSQL for the relational data, and ImmuDB as an append-only audit database (plus an immudb_states folder ImmuDB needs to boot consistently). Each had its own backup path writing to its own S3 prefix. A restore picked “latest dump” from each prefix. Two runs of “latest” can come from different backup executions: a PostgreSQL dump from 12:00 paired with an audit-DB backup from a partial 13:00 run. The restored state is internally inconsistent.

SSH/bastion is an attack surface I did not want. Operator day-2 (backups, restores, deploys) needed shell access. Bastions get phished, SSH keys leak, and audit logs live in places nobody reads.

This is a model for a single-host MVP:

• one EC2 host
• Docker Compose: PostgreSQL + ImmuDB
• cron-driven full backups on the host
• manual backup/restore runbooks triggered from GitLab CI over SSM
• no PITR, no RDS, no WAL shipping

That last line is the tradeoff. I revisit it in Tradeoffs. Within those constraints, two things can be made verifiable: deploys that are checksum-verified and reversible, and restores that cannot mix state across databases.

Release and restore design

Immutable release bundles

I added a mise task in CI that produces a release bundle: a bundle.tar.gz plus a manifest.json carrying the bundle’s SHA-256. I upload the bundle to a shared deploy-artifacts bucket.

The deploy flow runs on the host over SSM:

# host deploy steps (excerpt)
# download bundle.tar.gz and manifest.json from the deploy-artifacts bucket
sha256sum -c manifest.json # verify; mismatch aborts the deploy
mkdir -p "releases/$SHA"
tar -xzf bundle.tar.gz -C "releases/$SHA"
# atomically swap the current symlink to the new release
ln -sfn "releases/$SHA" current.new && mv -Tf current.new current
# rewrite the cron entry so scheduled backups follow the new current release path
crontab -u ec2-user "releases/$SHA/cron/backup.cron"

On-disk layout after a deploy:

# /home/ec2-user/deploy-artifacts/backend/
deploy-artifacts/backend/
├── current -> releases/<sha>/
├── releases/
│   ├── <sha-2026-06-24>/
│   └── <sha-2026-06-25>/
└── shared/
    ├── backups/
    ├── immudb_states/
    ├── logs/
    └── tmp/

shared/ survives release switches. logs/backup.log is shared state, so the log outlives any single release. I keep two release directories on disk at all times: the active one and the previous one. Rollback is ln -sfn ../releases/<prev-sha> current plus a cron rewrite, no re-download.

I made the scheduled backup no longer run from a checkout; it resolves through current:

# entry point executed from cron, as ec2-user
cd /home/ec2-user/deploy-artifacts/backend/current/backend
uv run --project ../infrastructure/scripts python scripts/backup_databases.py --env <env>

uv pins the Python toolchain per-release; the glue code comes from the currently deployed release, never from a stale host checkout.

Coherent backup-sets

Every backup I run (scheduled or manual) does this in order:

1. acquire a shared host maintenance lock (a host-level file lock, e.g. /tmp/db-maintenance.lock)
2. back up PostgreSQL, the audit DB, and the audit-DB state folder
3. upload each artifact to the backup bucket
4. write sets/<timestamp>.json only on success of all selected components

The lock is shared with restore. Backup and restore cannot overlap on the same host.

The manifest proves the components were captured together in one consistent run. It records:

• backup timestamp
• selected components
• S3 key per component
• artifact size
• SHA-256 checksum for file-based artifacts

S3 layout:

# s3://<DB_BACKUP_BUCKET>/
├── postgres/
│   └── 20260625_120000.dump
├── immudb/
│   └── 20260625_120000.backup
├── immudb-states/
│   └── 20260625_120000_immudb_states/
│       └── ...
└── sets/
    └── 20260625_120000.json

A sets/<timestamp>.json existing means “these artifacts are coherent: restore them as a group.”

Restore prefers coherent sets

I kept restore manual by design; it is triggered from GitLab CI over SSM. Artifact selection, in priority order:

1. explicit selectors from CI variables, if provided (artifact names, not full S3 paths)
2. a specific sets/<timestamp>.json, if a timestamp was requested and the manifest exists
3. the latest completed backup-set manifest that contains the required components
4. legacy per-component “latest object” selection, only when no suitable manifest exists yet

I kept the system backward-compatible with older per-component backups, but I made it avoid mixed restore inputs whenever a coherent set exists. Restore now also fails if a requested component backup is missing; silent partial restore is off the table.

Pre-restore safety backup + first-boot restore

Before any destructive restore, the host creates a fresh safety backup with a timestamp ending in _pre_restore. This is the immediate rollback point if the restore is interrupted, an incorrect artifact was chosen, or the new state is corrupt. The next backend:restore:matrix run can target the _pre_restore set.

On first boot, a fresh host pre-selects the latest coherent backend backup set (cloud-init user-data). I made it fall back to per-component latest only when necessary, and write a pending restore marker at /home/ec2-user/env.restore.pending.

The next backend deploy applies that pending restore automatically, before the application container starts. The marker can include pre-downloaded PostgreSQL and audit-DB artifacts, optional checksum/size metadata from the matching sets/<timestamp>.json, the matching audit-DB state folder selector, or an explicit skip marker.

One edge case bit me. ImmuDB’s immuadmin hot-backup exits successfully with zero bytes when the audit DB has no transactions yet. For a regular scheduled backup, an empty audit-DB artifact is a hard failure: the backup must produce real bytes. For the pre-restore safety backup, an empty audit-DB component is recorded as an explicit empty component instead of blocking the restore. The asymmetry is intentional: scheduled backups are strict; pre-restore safety backups are permissive, because blocking a safety backup defeats its purpose.

Keyless day-2 over SSM

No SSH. No bastion.

I declare the infrastructure and IAM in Terraform. The EC2 role carries AmazonSSMManagedInstanceCore: the minimal managed policy for SSM agent connectivity. AWS Systems Manager Run Command executes the deploy/backup/restore scripts on the host. GitLab OIDC authenticates the CI job to AWS with short-lived STS tokens; no long-lived keys in CI variables.

The host is Amazon Linux 2023. Two GitLab matrix jobs cover the admin surface:

• backend:backup:matrix
• backend:restore:matrix

Both invoke the admin entry point over SSM. Every backup and restore is observable as a CI job and an SSM command history entry.

Verification

Schedule is 2x/day: 00:00 and 12:00 Europe/Madrid. Artifact filenames are timestamped in UTC; that split is intentional. Madrid local midnight maps to roughly 23:xx UTC the previous day in winter, so the schedule triggers look offset by one day in the bucket listing. Operators reading s3 ls learn to think in UTC; the cron schedule is read in Madrid.

Tradeoffs

This is a cron-snapshot model, not point-in-time recovery. RTO is “spin up a fresh host + apply the latest coherent set.” RPO is bounded by the 2x/day schedule: up to twelve hours of committed writes can be lost between the last successful backup and a failure event. For a single-host MVP with low write volume, that was an acceptable tradeoff against the operational cost of standing up RDS or self-managed WAL archiving. It is not a substitute for PITR at scale. The next step for tighter recovery is RDS with automated backups and PITR, or managed Postgres with continuous WAL archiving. The immutable-release and coherent-set work would carry over to that world unchanged; the cron-snapshot backup layer would not.

ImmuDB hot-backup’s “no bytes when empty” behavior caused a debugging cycle on the safety-backup path. A successful exit with zero output looks like a bug, not a valid state. Special-casing empty-as-valid for the safety backup while keeping the strict non-empty check for scheduled backups took a while to land. Documenting that asymmetry up front (in the backup script’s docstring, not just the runbook) would have saved the effort.

References

• AWS Systems Manager: what it is. Run Command and Session Manager for keyless day-2 operations.
• AmazonSSMManagedInstanceCore managed policy. The minimal role for SSM agent connectivity.
• GitLab CI/CD with AWS via OIDC. Short-lived STS, no long-lived CI keys.
• cloud-init user-data. First-boot restore marker on fresh hosts.
• PostgreSQL pg_dump. The PostgreSQL backup mechanism.
• ImmuDB documentation. immuadmin hot-backup semantics, including the empty-backup edge case.
• mise. Task runner used to build the release bundle.
• uv. Pinned Python toolchain per release.
• Amazon Linux 2023. Host OS.
• Atomic symlink swap / blue-green deploy pattern. General prior art for the current -> releases/<sha> layout.