Skip to content
Fran Gonzalez
← Back to blog
(updated Jul 17, 2026)·Clanker·14 min read

Practical cache strategies for ephemeral Dagger CI

A decision guide to persistent engines, host-side cache bridges, registry proxies, and cold execution after Dagger 0.21 removed BuildKit remote export.

Development#dagger#caching#ci#github-actions#buildkit
Some matmuls wrote this slop, sorry. My goal with this content is to document some work I (a real human bean) do while poking the Clanker, and try to learn something along the way.

Dagger 0.21 leaves an ephemeral runner with a local cache that disappears when the engine exits. The useful question is which persistence boundary matches the workload and the amount of infrastructure I am willing to operate.

The companion source investigation, _EXPERIMENTAL_DAGGER_CACHE_CONFIG is inert in Dagger 0.21, traces why the former BuildKit bridge no longer exports that cache.

Note

Fact-checked on 2026-07-16 for Dagger 0.21.7 and GitHub Actions cache API v2. Options tied to hosted products, action versions, and Dagger’s planned exporter need re-verification before adoption.

Choosing a persistence boundary

Dagger 0.21’s DagQL cache needs a different persistence boundary. The options split into five groups: preserve the engine, bridge selected data, cache dependencies upstream, move selected work to another cache system, or reduce the amount of cold work.

OptionWhat survives across runsWorks on ephemeral GitHub-hosted runnersMain tradeoff
Dagger Cloud distributed cacheDagger-managed cache across enginesYesPaid distributed-cache service
Persistent engine on each self-hosted runnerDagQL results, filesystem snapshots, and cache volumes local to that runnerNoRunner affinity and persistent-disk operations
Shared remote Dagger EngineOne engine cache shared by several CI clientsYesNetwork, capacity, isolation, and availability management
Managed persistent runner or engineProvider-managed persistent Dagger cacheYesVendor cost and platform dependency
Engine-volume snapshotMost engine-local stateYes, through an archive bridgeLarge transfers and coupling to engine internals
Bidirectional GitHub Actions cache bridgepnpm store or another explicit directoryYesCustom Dagger inputs and outputs; Dagger operations still execute
S3, MinIO, or CI-native archive bridgeSelected stores or outputsYesCustom archive lifecycle, keys, and synchronization
Pull-through package registrynpm package tarballs and metadataYesService operation; package linking and scripts still execute
Published CI base imageToolchain, installed dependencies, and prepared filesystem layersYesImage build and tag lifecycle
Explicit function-output artifact cacheSelected generated directories or filesYesApplication-specific cache keys and restore logic
Buildx for selected stagesDockerfile and BuildKit layers through type=gha v2 or registry cacheYesTwo build graphs and cache models
Tool-native remote cacheCompiler or task outputsYesSeparate backend for each tool family
Vendor generated dependencies or outputsFiles committed with sourceYesRepository size, churn, and supply-chain review burden
Consolidate work into one Dagger engine sessionEngine-local reuse during the current jobCurrent job onlyCross-run graph stays cold
Accept cold cross-run executionNothingYesRepeated work in exchange for the smallest system
Wait for Dagger’s replacement exporterFuture native remote cacheEventuallyAvailability date and release version remain unspecified

Full Dagger-cache options

These options preserve the DagQL cache itself. They cover more than package downloads.

Dagger Cloud

Dagger Cloud is the direct match for the new cache architecture. It keeps the Dagger API and avoids workflow-specific cache plumbing. The distributed cache belongs to a paid plan, while the free service remains useful for traces.

A persistent engine on each self-hosted runner

A self-hosted runner can retain the Docker engine container and its volume between jobs. Each runner builds its own warm cache. Job affinity improves hit rates because a job sent to another runner sees that runner’s cache instead.

This setup is close to free when persistent self-hosted runners already exist. Disk sizing, garbage collection, runner replacement, and engine upgrades become operational concerns.

One shared remote engine

Ephemeral CI runners can connect to a long-running Dagger Engine on a VM, bare-metal host, or Kubernetes StatefulSet. Every client then reaches one local DagQL cache. The Dagger maintainers’ scaling discussion #6486 describes this stateful-engine model: local cache gives the highest hit ratio and lowest cache latency, while the always-on host adds cost and capacity planning.

A shared engine also introduces a trust boundary. Concurrent jobs share compute and cache infrastructure, so access control, workload isolation, noisy-neighbor limits, network latency, backups, and blue-green engine upgrades need explicit designs.

A managed persistent runner or engine

A managed runner can supply the persistence boundary without operating the VM directly. Depot’s Dagger integration is one example. This retains Dagger 0.21 and trades infrastructure ownership for provider cost and dependency.

Engine-volume snapshots

Exporting and restoring the Dagger engine’s Docker volume can carry local state between ephemeral runners. The original CI-cache discussion records /var/lib/dagger backup and restore as a community workaround.

A safe experiment would stop the engine before capture, pin the exact engine version, checksum the archive, and measure compression plus transfer time. A large engine cache can cost more to archive than the cold build costs to repeat. The storage layout is an internal implementation detail, so this remains a fragile workaround.

Data-bridge options

These options keep the Dagger 0.21 engine ephemeral and move only chosen data across the host boundary.

A bidirectional GitHub Actions cache bridge

The workflow can restore a pnpm store with actions/cache, pass that directory into the Dagger module, run the install, export the updated store back to the same host path, and let the action’s post step save it.

Both directions are required:

  1. actions/cache restores .cache/pnpm-store on the runner.
  2. A Dagger Directory argument imports that store into the container.
  3. pnpm install reads and updates /pnpm-store.
  4. A Dagger function returns the updated store as a Directory.
  5. dagger call ... export --path .cache/pnpm-store writes it to the runner.
  6. The cache action saves the host directory after the job.

A restore-only bridge stays stale because packages downloaded inside Dagger never reach the host cache.

As of July 16, 2026, the current action is v6.1.0 on Node 24. Its signed, full-SHA pin is:

- name: Restore pnpm store
  uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
  with:
    path: .cache/pnpm-store
    key: pnpm-${{ runner.os }}-${{ hashFiles('pnpm-lock.yaml') }}
    restore-keys: |
      pnpm-${{ runner.os }}-

The lockfile hash gives exact hits, and the prefix allows pnpm to reuse older package blobs after a dependency change. The bridge caches dependency content rather than DagQL results. Package linking, lifecycle scripts, and the Dagger install operation still run.

GitHub cache branch restrictions also apply. A pull-request run can read cache from its current branch, base branch, and default branch. Scope and key design should avoid one branch overwriting unrelated data.

An object-store or CI-native archive bridge

The same bidirectional pattern can use S3, MinIO, GitLab distributed cache, or another CI’s native cache instead of actions/cache. The workflow restores an archive, passes it into Dagger, exports the updated directory, and uploads the new archive.

This option is portable across CI providers and useful when object storage already exists. Compression time, archive size, concurrent writers, retention, corruption handling, and credentials become part of the cache design. Content-addressed package stores compress and transfer differently from ordinary build outputs, so measurements should decide whether the bridge pays for itself.

An explicit function-output artifact cache

A Dagger function can return a generated directory, compiled output, or other expensive result. The workflow can store that output under a key derived from its declared inputs, then pass it back as a Directory on the next run.

This is an application-level cache rather than a transparent engine cache. It works best for a small number of expensive and stable boundaries. Each boundary needs a complete cache key, schema/version marker, restore path, and invalidation rule.

GitHub Actions artifacts can also carry immutable outputs between jobs or workflows. They fit handoffs and retained build products better than a frequently updated dependency cache.

Upstream dependency options

These options make cold Dagger operations cheaper while leaving the DagQL graph ephemeral.

A pull-through package registry

Nexus Repository or Verdaccio can cache npm packages once and serve them to every runner. This reduces network transfer while keeping Dagger 0.21 unchanged. Package extraction, linking, and build scripts still run.

This option becomes more useful when several repositories, CI providers, or developers share the same dependencies. The companion post, Dependency caching belongs at the registry layer, not the package store, covers that boundary.

A Docker or OCI pull-through registry can do the same for base images. This shortens image pulls and centralizes upstream access, while Dagger still evaluates the operation graph on each fresh engine.

A published CI base image

A separate workflow can build an OCI image containing the mise-managed toolchain, pnpm store, node_modules, or another prepared dependency layer. Tagging the image with hashes of mise.toml, mise.lock, pnpm-lock.yaml, and relevant platform inputs makes invalidation explicit. Dagger 0.21 can start from that image and copy the changing source tree over it.

The registry then persists ordinary OCI layers without depending on Dagger’s remote-cache exporter. Buildx can create the image with type=gha v2 or registry cache. This approach moves dependency installation into image production, adds image publishing and cleanup, and requires careful handling of native dependencies and target architecture.

Mise can remain the version authority because the image build reads the committed mise configuration and lockfile. The generated image is an artifact of those inputs rather than a second hand-maintained version list.

Vendoring dependencies or generated outputs

Dependencies or generated outputs can live in the repository or Git LFS and arrive with checkout. This gives deterministic availability and removes a cache service from the runtime path. Repository growth, large diffs, platform-specific files, license review, security updates, and merge conflicts usually make this a narrow fit.

Alternate build-cache options

These options retain Dagger for orchestration and give selected stages to systems with working remote caches.

Buildx for selected stages

BuildKit supports type=gha v2 directly through docker buildx. A workflow can move an expensive Dockerfile build outside Dagger, use cache-from: type=gha and cache-to: type=gha,mode=max, then let Dagger handle the remaining orchestration.

A registry cache is another Buildx destination and works across GitHub Actions, GitLab CI, and local builders. This preserves remote layer caching for the selected stages and creates two build graphs in one pipeline.

Tool-native remote caches

Compilers and task runners may already support remote outputs: Gradle build cache, Bazel remote caching, sccache, ccache, Turborepo remote caching, Nx remote caching, and similar systems. Those caches can run inside Dagger containers because the tool owns the protocol and backend.

This option targets compiled or generated outputs rather than Dagger layers. It becomes useful in polyglot repositories where compilation dominates. The current Astro blog spends more time on dependency installation and checks, so a compiler cache offers little coverage here.

Workflow-only options

These options improve the current job and leave cross-run persistence unchanged.

Maximize one engine session

A GitHub Actions job can reuse one local Dagger Engine across calls. Combining checks and artifact export into one top-level Dagger function can also avoid repeated client startup and guarantee that the build result used for validation is the result exported to dist.

Within one module call, shared container construction and ordinary layers can be reused. Cache volumes also remain warm for the lifetime of that engine. This approach reduces duplicate work in one run and leaves the next GitHub-hosted runner cold.

Accept cold cross-run execution

A small public project can choose the simplest system: remove the inert cache configuration, keep Dagger 0.21, and pay the cold-build time. GitHub-hosted minutes are unlimited for this public repository, while developer attention and external services still carry costs.

This option becomes stronger when a pull-through registry already makes downloads fast or when the measured archive transfer takes as long as installation.

Wait for Dagger’s replacement exporter

The maintainer response in issue #13405 says an improved cache exporter is under development. Staying on 0.21 avoids migration work when that backend arrives. The public response leaves the release target and compatibility undefined. Current CI should treat the feature as unavailable.

Paths that 0.21 currently leaves unavailable

Three ideas appear in Dagger’s history but provide no current 0.21 solution:

  • _EXPERIMENTAL_DAGGER_CACHE_CONFIG remains parsed and inert.
  • Host-backed shared cache volumes from PR #7646 remain unmerged.
  • A shared filesystem mounted into several engines lacks the coordination and supported storage-driver path proposed in issue #8004.

Larger GitHub-hosted runners add CPU, memory, and disk for one job. Their ephemeral lifecycle still discards the local engine cache after the run.

Decision for this project

The realistic choices for this small public repository are a bidirectional host-side pnpm-store bridge, an existing pull-through registry, or accepting cold execution. A persistent engine preserves more of Dagger’s native cache, but its operational cost exceeds the current build-time savings.

I would measure archive transfer and install time before adding the host bridge. If the bridge does not beat a cold install consistently, accepting the cold graph is the smaller system while Dagger’s replacement exporter remains unavailable.

References

Dagger source and the 0.21 cutover

Dagger cache design history

Old BuildKit bridge limitations

GitHub Actions and alternatives

This post was written with AI assistance.