Skip to content
Fran Gonzalez
← Back to blog
·Clanker·8 min read

AWS auth in a Gondolin VM: why SigV4 breaks the secrets bridge

The Gondolin bridge substitutes placeholders in request headers, but the AWS SDK signs the request over the placeholder value. The substitution happens too late and every call returns SignatureDoesNotMatch. The fix is to mount the host's ~/.aws/ read-only instead.

Development#gondolin#aws#sigv4#secrets#sandboxing#identity-center
Some matmuls wrote this slop, sorry. My goal with this content is to document some work I (a real human bean) do while poking the Clanker, and try to learn something along the way.

I added AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_SESSION_TOKEN to gondolin.config.json the same way I had the LLM key, with one entry per AWS service and hosts pinned to the AWS endpoints. The launcher sourced the real values via pass-cli, the VM saw placeholders, and every call returned SignatureDoesNotMatch. I had assumed “key in a header” was enough. That assumption breaks when the rest of the request is signed over that header.

Symptom

The setup post on this setup has a “Tradeoffs” section that flags one auth shape to verify: “A provider that expects the API key inside a JSON body defeats Tier 2.” Header-based auth is presented as fine. I was extending the same setup to a monorepo where the agent needs to make AWS calls from inside the VM, and I treated the AWS SigV4 triplet like any other header-based secret. With SigV4, the SDK derives a signing key from the secret access key and signs the canonical request, which lists the session token (x-amz-security-token) among its signed headers. The signature is then attached as the Authorization header; the secret access key never travels in a header at all. The bridge can swap the session-token header value at egress, but the SDK has already signed over the placeholder token. AWS recomputes the signature from the real token now in the header, the two never match, and the call dies with SignatureDoesNotMatch.

The empirical proof is one aws --debug sts get-caller-identity away. The dump below shows the placeholder baked into the canonical request and into the signature, with the bridge yet to substitute on the way out:

CanonicalRequest:
POST
/

content-type:application/x-www-form-urlencoded; charset=utf-8
host:sts.eu-west-1.amazonaws.com
x-amz-date:20260622T161417Z
x-amz-security-token:GONDOLIN_SECRET_d77bfb82814175ffebe942dae2f19b9b07c19d2a126388ba

content-type;host;x-amz-date;x-amz-security-token
ab821ae955788b0e33ebd34c208442ccfc2d406e2edc5e7a39bd6458fbb4f843

StringToSign:
AWS4-HMAC-SHA256
20260622T161417Z
20260622/eu-west-1/sts/aws4_request
0e94f5a7bfca0b5ec5b9ba2a092ea753dcfc4dd87b425d337628d2fe23ff19aa

Signature:
ca337bc2bf3cc9ec473e2808fe48a0d041fed3eae57165b6224b683cef3fc0d0

The bridge’s secrets model substitutes placeholders in request headers at the last possible moment, just before the host forwards the upstream request. That works for any provider that puts the secret in a header and does not derive a cryptographic signature over the rest of the request from the secret. AWS SigV4 does. The Gondolin docs warn against request bodies (substitution never touches bodies) and against mounting host secret files (a separate ~/.aws warning in the operational guidance). They do not call out signed-headers auth. That is the gap.

Investigation and fix

The two real options are bearer tokens (the small set of AWS services that accept Authorization: Bearer <token>, like API Gateway and AppSync; the SSM/S3/ECR/STS services the monorepo uses require SigV4) or mounting the host’s ~/.aws/ read-only into the VM at /root/.aws/. The AWS SDK in the VM reads the real session and refresh tokens from the cache, signs the request with the real key, and the host’s network stack carries it. The Gondolin bridge is not in the signing chain. The mount is what I went with, as a per-repo opt-in in the launcher:

// ~/.config/gondolin/launcher/gondolin-run.ts
type GondolinConfig = {
  // ... existing fields ...
  mountHostAwsConfig?: boolean; // opt-in: mount ~/.aws into the VM at /root/.aws (RO)
};

// in main(), after the glab-cli mount
if (config.mountHostAwsConfig === true) {
  const awsDir = path.join(process.env.HOME!, ".aws");
  if (fs.existsSync(awsDir)) {
    mounts["/root/.aws"] = new RealFSProvider(awsDir);
    console.error(
      `[gondolin-run] aws mount: ${awsDir} → /root/.aws (host RO, opt-in)`,
    );
    console.error(
      `[gondolin-run] aws mount: blast radius is up to 12h (aws login refresh window)`,
    );
    console.error(
      `[gondolin-run] aws mount: revoke with 'aws logout', audit at https://console.aws.amazon.com/cloudtrail/`,
    );
  }
}

In the repo’s gondolin.config.json, the opt-in is one line, and the AWS_* entries come out of secrets and out of allowHosts:

{
  "imageDir": "/Users/fran/.local/share/gondolin-images/pi-vm-dev",
  "mountHostAwsConfig": true,
  // ... shadow / shadowTmpfs / allowHosts (no AWS endpoints) / secrets (no AWS_*)
}

Default is false. Repos that do not need AWS from inside the VM get nothing.

Verification

The validation ran the launcher with the new opt-in and aws sts get-caller-identity from inside the VM. The result:

  • /root/.aws/config and the aws login session tokens in /root/.aws/{login,cli}/cache/ are reachable in the VM
  • aws sts get-caller-identity returns the host’s IAM identity, with the same role chain the host uses
  • Reading an SSM parameter from inside the VM returns the same value the host gets, confirming the mounted credentials reach Parameter Store
  • The 17/17 tool smoke test for the image still passes, with the new layer on top: mountHostAwsConfig: true sits beside the existing glab-cli mount, and both are best-effort (skipped if the host directory does not exist)

Preconditions and blast radius

The mount is only safe when the host’s AWS auth is short-lived. The precondition check before enabling the opt-in:

# Long-lived keys on disk? If a credentials file has IAM user keys, STOP.
[ -f ~/.aws/credentials ] && { echo "DANGER"; exit 1; }

# aws login session cache populated?
ls -la ~/.aws/login/cache/ ~/.aws/cli/cache/

# MFA required on the AssumeRole trust policies?
grep -r MultiFactorAuthPresent ~/.aws/ 2>/dev/null

When the host uses aws login ([aws-cli 2.32+]1) for its AWS access, you sign in once with your AWS Management Console credentials (username and password) in a browser. The CLI obtains temporary credentials, caches them in ~/.aws/{login,cli}/cache/, and leaves no long-lived ~/.aws/credentials file on disk. The CLI auto-refreshes these credentials for up to 12 hours2; after that window, a fresh browser sign-in is required. The blast radius if the VM is compromised is bounded by that 12-hour refresh window and is revocable with aws logout, which clears the cached credentials and stops the refresh3. Every API call is also in CloudTrail, so the audit trail is independent of what the agent does locally.

For the “oh shit” recovery, a small host-side helper:

# ~/.config/gondolin/aws-revoke.sh
aws logout     # ends the local session and stops credential refresh (kills the in-VM agent's access)
# print the CloudTrail console URL to audit recent API calls from the host identity

It prompts before running, runs aws logout, prints the CloudTrail console URL for audit, and exits non-zero if aws logout is unavailable. It is the single recovery action to remember under stress.

Security tradeoffs

The setup post’s “Tradeoffs” section calls out the JSON-body case but does not mention signed-headers. That section should grow a second bullet for signed-headers auth, and probably a generic note that “any auth where the request itself is signed over the secret value” is a no-go. That edit is the right follow-up to this post.

I would also have built the recovery helper from day one. The 30 seconds it takes to set up pays for itself the first time you actually need it.

References

Footnotes

  1. The aws login command ships in the AWS CLI as a feature:credentials entry under the 2.32.0 release heading, not 2.16. Confirmed against the official AWS CLI v2 CHANGELOG and the aws login command reference. ↩

  2. The CLI auto-refreshes aws login credentials for up to 12 hours from a refresh token obtained at sign-in; after that window, a fresh browser sign-in is required. Stated in the AWS CLI sign-in guide and covered in the AWS Security blog post on aws login. ↩

  3. aws logout clears the cached credentials and stops the refresh. Because the refresh token only enables rotation within the 12-hour window, the maximum exposure from a compromised VM is bounded by the time remaining in that window. ↩

This post was written with AI assistance.