From a 6.2GB baked toolchain image to boot-time installs from repo lockfiles

My Gondolin dev VM image (earlier setup), which runs the pi coding agent inside a micro-VM, carried a 6.2GB toolchain baked in from one monorepo (java, maven, terraform, trivy among others). Other repos mounted on the same image used a fraction of it. This post records why I removed the baked toolchain and how each repo now installs its own tools at boot.

What the baked image cost

The original Dockerfile copied one monorepo’s mise.toml files as a build context, ran mise install during docker build, and shipped the installed tools in the image. That gave instant boots and zero tool egress. Four problems came with it, all consequences of the image knowing about one repo:

• The build was coupled to one repo. The image consumed a snapshot of that monorepo’s mise configs and asserted its exact stack in the build’s smoke checks (java -version, mvn --version, terraform --version). Building against a different repo failed on java -version, a tool the second repo never declared.
• The image ran on one architecture. The committed mise.lock had no linux-arm64 entries, so the Dockerfile re-locked for arm64 inside the build and deleted firebase-tools from the config before installing, because that tool publishes no arm64 linux binary.¹ My x86_64 server could not use the image at all.
• A global tool declaration leaked into every workspace. The image’s global mise config declared a tool (npm:renovate) that belonged to the one monorepo. Every mounted workspace inherited the declaration, and repos using lockfile = true then failed their locked install because their own lockfile did not contain that tool.
• Tool versions were pinned in two places. The image carried one set of versions; each repo’s lockfile carried another. They could disagree, and did when the image drifted from the repos between rebuilds.

The repos already describe their toolchain completely: mise.toml plus mise.lock pins exact versions with per-platform URLs. Duplicating a subset of that in an image added the four problems above without adding information.

The redesign: the repo installs its tools at boot

The image now provides the runtime only: Debian slim, mise, node 26, npm 12, pi, gh, and glab. It builds natively on arm64 and x86_64 (a TARGETARCH build arg maps Docker’s arch names to the vendor-specific ones mise and node use). Gondolin images are local artifacts per host rather than registry manifests, so multi-arch support here means the Dockerfile must build on each host, which it now does. The image is 1.74GB, down from 6.17GB.

The launcher does three new things at boot:

// ~/.dotfiles/.config/gondolin/launcher/gondolin-run.ts

// 1. Mount a host-side tool cache (RW). mise verifies checksums
//    against the workspace lockfile, so cached downloads are safe
//    to reuse across boots:
mounts["/var/cache/mise"] = new RealFSProvider(toolCache);

// 2. After seeding pi's settings, install the workspace toolchain
//    from the repo's own mise.toml + mise.lock:
await vm.exec("if [ -f mise.toml ]; then mise install; fi", {
  cwd: "/workspace",
});

// 3. Append common tool-download egress hosts for dev images only:
config.allowHosts.push(...DEV_TOOL_EGRESS); // github, npm, pypi, nodejs, ...

Two repo-side changes made the boot install work on every machine I run:

One lockfile covering three platforms. mise lock locks the current platform by default. Running mise lock --platform linux-arm64 --platform linux-x64 on a mac produced a single lockfile with 18 arm64, 19 x64, and 19 macos entries. The mac, the x86_64 server, and the arm64 VM all resolve from the same committed file.

Platform-scoped declarations for tools without arm64 assets. Keeping firebase-tools in the root [tools] made every locked arm64 install fail with “No lockfile URL found for firebase on platform linux-arm64”. I first tried an explicit [tools.firebase.platforms] block; that block is an asset-pattern map for the aqua backend, and mise still attempted the install on arm64 (observed, not documented; the configuration docs describe the working mechanism). The fix was mise’s OS- and arch-specific config files: mise.macos.toml and mise.linux-x64.toml each declare firebase-tools, and arm64 hosts never load either file.

Per-repo network policy moved into each repo as well: a gondolin.config.json with that repo’s allowHosts, shadow rules, and image selection, with a minimal global fallback for repos without one. The launcher appends the common tool-download hosts automatically, so no repo config repeats them.

Measured results

The boot timings come from the same 21-tool workspace before and after the change; the cold figure is the first boot on an empty cache.

Tradeoffs I accepted

• Cold boots download the toolchain. A first boot on an empty cache takes about 2.5 minutes for that workspace; warm boots with the host cache take 74 seconds. The baked image booted instantly. For workloads that boot VMs very frequently and discard them, baking would still be the better trade.
• The egress allowlist grew. Locked installs reach github release assets, npm, pypi, nodejs.org, apache, and hashicorp. The VM holds no secrets beyond an api-scoped placeholder key, so this widens download targets without widening secret exposure. It is still a larger allowlist than before, and it is now part of the setup’s threat model.
• The old image is gone. I kept no dev-image rollback tag; the previous Dockerfile remains in the dotfiles git history if the baked approach is ever needed again.

References

• Gondolin sandboxing for AI coding agents: two tiers from tool-only to full clean room. The earlier post describing the pi-secure and pi-secure-vm tiers this image serves.
• Gondolin Documentation. VFS providers, the VM SDK, and image builds.
• mise lock and mise.lock lockfiles. The lock command and cross-platform lockfile semantics.
• mise configuration. The OS- and arch-specific config files used to scope firebase-tools.
• pi. The coding agent that runs inside the VM.