Fixing a 45-minute CI hang in Tauri WebDriver native tests

A Debian release job on GitHub Actions exceeded its 45-minute limit. The test inside it had already failed 26 minutes earlier. Fixing the hang meant first fixing how I investigated it.

Symptom

DeepBlock is a Rust/Tauri localhost time-blocking app. Its release workflow runs quality gates, builds signed macOS packages and a Debian package, installs the .deb on clean Debian 12 and 13 containers, and assembles a draft release. The Debian job also runs a native WebKit UI test following the Tauri WebDriver testing guide through tauri-driver, the WebDriver intermediary that wraps Linux’s WebKitWebDriver.

The run that finally cleared quality gates, macOS packaging, and Debian packaging died at the native test:

18:50:12  ✖ desktop runtime loads the planner and help routes (61222.61808ms)
18:50:13  COMMAND deleteSession()
19:16:36  ##[error]The operation was canceled.

The test failed in 61 seconds. Teardown then sat silent for 26 minutes until the job timeout killed the run, with no Linux artifacts uploaded.

Investigation

That was run six. The five before it, all the same afternoon, each failed somewhere different, and I treated each full run as the next experiment:

• Run 1: Cargo lock timeout. Concurrent Cargo quality tasks (clippy, test, build) contended on build-directory locks. Fixed by serializing the task list.
• Runs 2 and 3: disk exhaustion. No space left on device while compiling libdeepblock_desktop.a. The runners had roughly 14 GB free after tool setup; a debug build plus bundling intermediates exceeded that. I disabled incremental compilation and trimmed debug symbols, which bought headroom without a pass.
• Run 4: two failures at once. macOS packaging failed with lipo: unknown architecture specification flag. I assumed thin/fat binary handling and spent time in artifact assembly code; the actual cause was Xcode lipo’s argument order in three call sites of my collection script. The Debian job hit disk exhaustion again, so I reclaimed about 25 GB of preinstalled SDKs (/usr/local/lib/android, /usr/share/dotnet, /opt/ghcup), a pattern I later recognized in Firezone’s release setup.
• Run 5: APT stall. apt-get update hung on the default Ubuntu mirrors and the job was cancelled; I saw the same stall twice across runs. I rebuilt all APT usage into a bounded script (retries, 30s HTTP timeouts, official HTTPS mirrors, retained .debs, SHA-pinned cache), modeled on Firezone’s apt-install action.
• Run 6: the 45-minute timeout. The subject of this post.

Root cause

Two defects, both visible in the log above.

A case-sensitive assertion against CSS-transformed text. The test waited for DeepBlock desktop in document.body.innerText. The badge carries a Tailwind uppercase class, so WebKit reported DEEPBLOCK DESKTOP. The app worked; the assertion was wrong.

Unbounded teardown. test.after awaited client.deleteSession() before killing the driver process. WebdriverIO documents a 120-second request timeout and three retries by default.¹ Those settings allow multi-minute requests; they alone do not explain the entire 26-minute silence. The log shows teardown entered session deletion and never reached process cleanup before cancellation. I checked those defaults only after the hang; I had written cleanup assuming session deletion would be quick.

Fix

The harness gained three properties it lacked: correct matching, bounded failure, and no duplicate work.

Body assertions became case-insensitive, because the UI intentionally uppercases via CSS:

// apps/desktop/ui/scripts/tauri-e2e-core.ts
export function containsBodyText(
  text: unknown,
  fragment: string,
): text is string {
  return (
    typeof text === "string" &&
    text.toLowerCase().includes(fragment.toLowerCase())
  );
}

In apps/desktop/ui/scripts/tauri-e2e-core.ts, teardown now treats session deletion as best effort, then terminates the driver’s process group. A three-second Promise.race caps deleteSession(), followed by SIGTERM, a one-second grace, SIGKILL, and fixture removal. On Linux, process.kill(-pid) signals the process group, including WebKitWebDriver. Cleanup ignores an already-exited group (ESRCH) and uses nested finally blocks so a cleanup error cannot skip fixture removal.

The WebdriverIO client now connects with connectionRetryTimeout: 10_000, connectionRetryCount: 0, logLevel: 'error', and every node:test call carries an explicit timeout. The runner subprocess gets a four-minute ceiling with killSignal: 'SIGKILL', and the workflow step a five-minute timeout-minutes.

The workflow also changed shape. The native test previously rebuilt the app in a debug profile, another ~4 GB of intermediates and 4 minutes of compilation, which had motivated the disk cleanup steps in earlier runs. The release job had already built the release binary, so the test now reuses it via a DEEPBLOCK_TAURI_E2E_APP variable pointing at target/x86_64-unknown-linux-gnu/release/deepblock-desktop. The build intermediates, the cargo clean, and the profile gymnastics are all gone. Linux packages are uploaded before the native test runs, so a test failure preserves diagnostic artifacts while still blocking installation, assembly, and the draft.

One more wrinkle: release runs check out an immutable tag, so harness fixes on a branch would never apply to them. The workflow now sparse-checkouts the native test tooling at github.workflow_sha and runs it against the tagged source’s dependencies and compiled binary. Harness fixes are test-only changes; rebuilds of old tags receive them.

Verification

• 53 helper regression tests now cover the release tooling, including a real SIGTERM-ignoring child process, smoke-fixture ownership, and rejection of early desktop exits.
• A real Debian 12 ARM64 container loads the planner and help routes through WebKit in ~1.7 s, including from a separately checked-out test-tooling directory.
• A deliberately failing assertion (waitForBodyText('Deliberately absent marker', 250)) must exit nonzero within 20 seconds; it exits in about 2.
• One explicitly approved candidate run then exercised the exact AMD64/CrabNebula CI path. The native step passed in five seconds. The Debian build/native job finished in 13m29s, within its existing 45-minute limit.

The local ARM64 check used upstream tauri-driver 2.0.4 via a PATH shim because the CrabNebula npm package lacked a Linux ARM64 build. It also first failed because macOS tar had smuggled AppleDouble ._*.json files into the source copy. The subsequent AMD64 CI result established the production harness combination.

The next failure was in the installation fixture

That candidate still failed overall. Debian 13 installed the CLI/server packages, then the server exited with Permission denied; fail-fast cancelled Debian 12. The smoke script created final directories with install -d -o smoke -g smoke, leaving an intermediate XDG_DATA_HOME/deepblock directory root-owned. I reproduced the failed write as the smoke user, listed every directory level explicitly, and added a regression guard.

This time I reused the retained AMD64 packages. I installed act through mise at version 0.2.89 and replayed the full smoke script in digest-pinned Debian 12 and 13 containers. Both passed in approximately 1m53s and 1m42s. The checks covered server/CLI operations, a 15-second desktop survival interval, coexistence, reinstall/removal, and preserved user-data hashes. I also bounded desktop termination with a two-second forced-kill grace and rejected early kills.

The first two local replay attempts failed before product testing because my bind mounts were absent. In act 0.2.89, a job’s container.options supersedes global container options, so I moved the mounts into the job definition. Reproduction tooling needed debugging too.

These replays used AMD64 emulation on Apple Silicon, read-only package mounts, no GitHub credentials, and an external deadline. act’s documented limitations include ignored permissions, environments, and job timeouts. The results establish the installation-smoke stage; signing, GitHub gate enforcement, and a complete assembled draft remain separate checks.

Follow-up

The biggest mistake was process: I used full release runs as a discovery loop. The first six runs took up to 66 minutes each, all in one afternoon, and I repeatedly relied on the next full run to validate a fix. Fran reported that roughly 700 of the month’s 3,000 included CI minutes had already been consumed. After the harness failure I changed the approach:

1. Read the raw logs before hypothesizing. The 61-second failure plus 26-minute silence sat in run 6’s log the whole time.
2. Reproduce the failing stage in isolation before patching it. A Debian 12 Docker container reproduced both native-test defects on demand, in seconds, for free.
3. Write the regression test first, then the fix.

I would also have checked the WebdriverIO retry defaults before writing cleanup code that assumed fast session deletion. Bounded teardown has to be a design property of the harness, and the defaults do not provide it.

References

• Tauri WebDriver testing guide. The two-port tauri-driver architecture these tests drive.
• tauri-driver crate. The upstream intermediary driver; 2.0.4 is what local ARM64 validation used.
• WebdriverIO options. The retry defaults that made unbounded teardown hang.
• Node.js test runner. Per-test timeouts used throughout the harness.
• Node.js process signals. Process-group cleanup on Linux.
• act and unsupported functionality. Local Linux-stage replay and its limits.
• mise exec. Pinned local act installation/execution.
• GitHub Actions timeout-minutes. The step-level bound on the native test.
• Firezone. Reference for bounded, cached APT and SDK reclamation on hosted runners.