Why Lychee checked 336 links in CI and 6,633 locally

The link check in GitHub Actions passed with 336 links. The same Lychee configuration found 6,633 links locally and reported broken URLs. The difference came from one pair of shell quotes around ./dist/**/*.html.

The suspiciously small green check

This site’s Dagger CI workflow builds Astro inside Dagger, exports dist/ to the GitHub runner, and passes the generated HTML to lychee-action:

# .github/workflows/ci.yml
- name: Check links
  uses: lycheeverse/lychee-action@e7477775783ea5526144ba13e8db5eec57747ce8 # v2.9.0
  with:
    token: ${{ secrets.GITHUB_TOKEN }}
    args: >-
      --verbose --no-progress --root-dir dist
      --exclude 'vm70\.neocities\.org'
      ./dist/**/*.html
      ./README.md

The successful workflow run produced this summary:

Total       336
Unique      174
Successful  314
Excluded     22
Errors        0

A production build contained 189 HTML files. A check that read only 336 links across that site looked too small, especially because most blog posts contain a References section.

The local mise task used the same options and appeared to mirror CI:

# mise.toml
[tasks.lychee]
description = "Check links in the built site (run `mise run build` first); mirrors CI"
run = "lychee --verbose --no-progress --root-dir dist --exclude 'vm70\\.neocities\\.org' './dist/**/*.html' ./README.md"

That run produced a different result:

Total   6633
Unique  1694
Errors    25

The local result included every generated post page. The CI result mostly covered top-level indexes.

Investigation: count files before links

I reduced the difference to the input paths before looking at HTTP behavior.

# What Bash expands without recursive globstar support
printf "%s\n" ./dist/**/*.html | wc -l
# 7

# What Astro generated
find dist -type f -name '*.html' | wc -l
# 189

In Bash, ** only becomes recursive when the globstar shell option is available and enabled. Otherwise the two stars behave like ordinary wildcard components. The workflow argument expanded to HTML files one directory below dist/, such as dist/blog/index.html and dist/tags/index.html. It skipped deeper paths such as dist/blog/<slug>/index.html.

I reproduced the CI count locally by passing the glob unquoted through Bash:

/bin/bash -c \
  "lychee --no-progress --root-dir dist \
  --exclude 'vm70\\.neocities\\.org' \
  ./dist/**/*.html ./README.md"

The result matched CI exactly:

336 Total, 174 Unique, 314 OK, 0 Errors, 22 Excluded, 1 Redirect

This established the failure boundary. Astro exported the complete site. Dagger wrote all 189 HTML files to the runner. The shell reduced Lychee’s input set before link checking began.

Why the mise task reached the full tree

The mise command wrapped the glob in single quotes:

'./dist/**/*.html'

Those quotes preserve the glob as one argument. Lychee receives the pattern and performs its own recursive file matching. The GitHub Actions argument lacked those quotes, so the action’s shell expanded it first.

The commands looked equivalent in configuration review because both contained dist/**/*.html. Their execution boundaries differed:

The most robust input is the directory. Lychee’s documented lychee . form recursively checks supported files under a directory. Passing ./dist removes shell glob semantics from the path entirely.

A full local verification

I tested the directory input in offline mode first. Offline mode checks local file targets while blocking network requests, which separates deterministic site-structure failures from external HTTP behavior:

lychee \
  --no-progress \
  --offline \
  --root-dir dist \
  ./dist \
  ./README.md

The directory scan reached 6,839 links and found two missing local targets. Both came from a published Gondolin post linking to an AWS-auth post that was still a draft. Production correctly excluded the draft page, so the generated link pointed at a file that did not exist.

Total   6839
Unique  1699
Errors     2

That is useful separation:

• Offline mode validates root-relative links, generated routes, assets, and fragments against dist/.
• Online mode adds external redirects, moved documentation, rate limits, and server-side blocking.

The full online directory scan also surfaced external 404, 403, and 429 responses. I updated moved documentation links, replaced npm package-page links that block automated clients with their upstream GitHub repositories, and replaced two rate-limited GNU manual links with equivalent man7 references. I also removed the published link to the draft AWS-auth post. After those corrections, the full online run completed with zero errors.

Root-relative links require the built site

This repository cannot run a useful link check directly against Markdown alone. Blog posts use root-relative links such as:

[Previous post](/blog/dagger-ci-astro-blog)

Lychee needs the generated output and --root-dir dist to resolve that path to:

dist/blog/dagger-ci-astro-blog/index.html

This is why Lychee stays out of the pre-commit hook. The pre-commit checks operate on source files. Link validation operates on the production artifact after Astro excludes drafts and creates final routes.

The current commands also use .lycheeignore:

^http://localhost:3000$
^https://frangonf\.com/

The production origin is excluded because internal routes are checked against the local dist/ tree. Re-requesting the deployed site would duplicate those checks and make local validation depend on the currently deployed revision.

CI and local parity need a shared boundary

mise run ci:dagger runs the Dagger checks, including the Astro build inside its container. It does not export dist/ or invoke Lychee. GitHub Actions performs those as later workflow steps.

The local Lychee task also expects dist/ to exist already. Its description says it mirrors CI, but parity currently requires two commands:

mise run build
mise run lychee

I changed the shared setup around four explicit rules:

1. Both environments build the production site first.
2. Both pass ./dist and ./README.md as directory or file inputs.
3. Stable options live in one root lychee.toml.
4. CI and mise run Lychee 0.24.2.

The shared lychee.toml encodes a clear-errors policy (explained later):

# lychee.toml
root_dir = "dist"
cache = true
accept_timeouts = true
accept = ["100..=103", "200..=299", "403", "429", "500..=599"]
user_agent = "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36"

The callers now contain only execution-specific wiring:

# GitHub Actions
with:
  token: ${{ secrets.GITHUB_TOKEN }}
  lycheeVersion: v0.24.2
  args: ./dist ./README.md

# mise.toml
[tasks.lychee]
depends = ["build"]
run = "lychee --verbose --no-progress ./dist ./README.md"

The action token gives Lychee authenticated GitHub API capacity when checking GitHub links. Local runs can use GITHUB_TOKEN or --github-token for the same reason. The mise task now depends on build, so one command creates and checks the production artifact.

What the green check proved

The original CI check proved that 336 links in seven shallow HTML files were healthy. It did not establish link health for the generated blog posts.

The discrepancy was visible in the totals before any link-level debugging. Comparing input-file counts and Lychee summaries turned a green check into a coverage test:

I replaced the recursive glob with ./dist, moved stable policy into lychee.toml, pinned Lychee 0.24.2 in the action, and made the mise task build before checking. The local command now completes in one invocation:

mise run lychee

The first clean run checked 6,835 links with zero errors. A later run after more content was added checked 6,932 links with zero errors. The changing total is expected; the invariant is recursive coverage with no rejected links. The CI workflow now receives the same directory inputs and configuration instead of relying on shell glob expansion.

Full coverage exposed a second problem

Recursive coverage closed the silent gap and opened a noisy one. The first CI run after the fix failed on four external links that returned 200 in a browser: a Substack guide returned 403, the SLSA spec dropped the connection, and the Vite and TanStack docs timed out. The shared cause was the GitHub-hosted runner’s datacenter IP. CDNs and WAFs throttle datacenter ranges harder than residential IPs, so a URL that loaded in my browser timed out or refused the connection from the runner.

My first response was to exclude each offender. That fixed one run, then the next failed on a different site, then the one after on another. Excluding domains one at a time is whack-a-mole, and every run can surface a new site the runner cannot reach.

A blocklist was the wrong shape for the problem. A link is clearly broken only when the server reports the resource gone or invalid (definitive 4xx such as 400, 404, 410, 451). Everything else describes the checking environment: timeouts, connection failures, 403 bot-blocking, 429 rate-limits, and transient 5xx server errors. The lychee.toml above encodes that split with three documented options:

• accept_timeouts = true: a timeout is network or rate-limit flakiness, accepted as non-fatal.
• accept = ["100..=103", "200..=299", "403", "429", "500..=599"]: the first two ranges are lychee’s default success codes; 403, 429, and 5xx are added because they reflect the runner, not the link.
• user_agent = "…": a browser user agent is the documented remedy for Cloudflare and bot-detection connection failures, which return no status code and therefore have no accept entry.

Lychee’s network-errors guide drove the choice. Its retry-policy table already retries transient classes (5xx, 408, 429, timeouts) and treats definitive 4xx and connection-refused, firewall, and DNS failures as permanent.

The policy could not gate the deploy on its own. A later run failed on typescript-eslint.io (connection reset by peer) alongside the SLSA spec (timeout). Connection failures return no HTTP status code, so accept cannot cover them, and accept_timeouts only succeeds when timeouts are the sole problem. The browser user agent reduced the count but could not guarantee a connection the runner’s IP was never allowed to establish. lychee has no documented flag to accept connection errors, so they pass straight through to a failing exit code.

The reliable gate separates the two concerns. Offline mode (lychee --offline) checks only local targets against dist/ with no network: root-relative links, fragments, assets, and generated pages. It is deterministic, finishes in under 50 milliseconds, and a missing page or broken internal link fails the build there. The online check runs alongside it as informational (fail: false): it scans every external link and writes the moved or broken ones to the job summary for review, but a CDN dropping the runner no longer blocks the deploy. Internal integrity gates; external rot gets reported. Locally, mise run lychee:offline reproduces the offline gate and mise run lychee runs the full online check; both keep their exit codes, so the boundary holds in development.

References

• Lychee. Link checker documentation, recursive directory inputs, offline mode, caching, and GitHub authentication.
• Lychee configuration reference. Documented options for accept, accept_timeouts, user_agent, and retries.
• Lychee network-errors guide. Retry policy and the user-agent remedy for connection failures.
• lychee-action. GitHub Actions wrapper used by the CI workflow.
• Bash filename expansion. Shell wildcard expansion rules.
• Bash globstar. Shell option that gives ** recursive behavior.
• mise tasks. Local task definitions and dependencies.
• Dagger CI for this Astro blog. The pipeline that exports dist/ before Lychee runs.
• GitHub Actions run 29514692929. The observed 336-link CI result.