Skip to content
Fran Gonzalez
← Back to blog
·Clanker·15 min read

Clearing thirty Renovate merge requests in one pass

A retrospective of clearing a 30+ MR dependency backlog in a repository with JavaScript, Java, Python, and Terraform projects, including CI, supply-chain, and typecheck changes.

Some matmuls wrote this slop, sorry. My goal with this content is to document some work I (a real human bean) do while poking the Clanker, and try to learn something along the way.

Over two days I cleared a backlog of roughly thirty Renovate dependency merge requests on a repository containing JavaScript, Java, Python, and Terraform projects. The work also included a credential-free Terraform gate, three CI quality gates, a supply-chain patch, a backend static-analysis cascade, and a frontend typecheck regression.

This is a detailed log of what I did, what broke, and the resulting changes across dependency management, CI, and project tooling.

Note

This is a point-in-time account of one dependency pass. The versions, runner count, and CI job names are the values from that pass.

The lay of the land

The repository contains pnpm workspaces at the root for the transport apps and the end-to-end tests, four standalone pnpm workspaces (frontend-nuxt, admin-nuxt, web-app, and cms-app), a Java backend built with Maven, and Terraform infrastructure with ten stack roots and six reusable modules. Two self-hosted GitLab runners serve the repository.

The two-runner limit affected the processing order. Every merge triggers a main-branch pipeline. Leaving that pipeline running delays validation for subsequent MRs, so I canceled it after each merge.

CI lives in per-domain files under ci/projects/, included from .gitlab-ci.yml, each with changes:-gated rules. The local source of truth is one mise task, check:all, which runs the credential-free gates in sequence:

check:toolchain-versions -> check:terraform -> lint:all -> check:python
  -> backend:compile -> 7 typechecks

The seven typechecks cover api-service, mobile-app, frontend-nuxt, admin-nuxt, web-app, cms-app, and e2e-tests. The expected relationship was: if check:all is green locally, the MR should pass CI. This pass identified the cases where the local and CI checks differed.

Processing the dependency wave

The backlog was the standard Renovate output: patch and minor bumps, a handful of majors, and Terraform provider bumps. Renovate runs on a daily schedule (02:00 UTC). I processed them one at a time: verify CI, merge, cancel the main pipeline, repeat.

Two refresh paths emerged, and selecting the applicable one was the first recurring decision.

Non-lockfile MRs rebase cleanly. A single version bump in a package.json with no lockfile implications, or a Maven dependency, just needs glab mr rebase and CI re-runs against current main.

Lockfile MRs are different. Renovate regenerates the lockfile when it creates the branch, but by the time the MR reaches the front of the queue, main has moved. After rebase the committed lockfile no longer satisfies the bumped manifest, and pnpm install --frozen-lockfile refuses. The repeatable recovery is local:

git checkout renovate/some-branch
git reset --hard origin/main
# reapply only the intended version bumps (edit package.json / pnpm-workspace.yaml)
pnpm install          # regenerate the lockfile against current main
git add -A
git commit -m "chore(deps): bump X"
git push --force-with-lease

Three lessons live inside that flow.

First, Renovate branches are snapshots. The branch reflects the main it saw at creation time, not the main it will merge into. Any state that depends on the rest of the tree (lockfiles, generated code, cross-file constraints) goes stale. Call this the snapshot problem.

Second, git restore . after git reset --mixed main can discard changes left in the working tree. A mixed reset moves the branch pointer, resets the index to the target commit, and leaves the changes unstaged in the working tree. git restore . then restores the working tree from that index. I lost a rebase to this once. I then used reset --hard and reapplied the intended version changes from scratch.

Third, the lockfile-sync guard reads the committed tree (HEAD), not the working directory. A locally-correct lockfile regeneration is invisible to CI until it is committed. The local command can pass as soon as the file is written, while CI continues to read the previous committed file.

Supply-chain posture

Dependency updates in this repository use frozen lockfiles, build-script allowlists (allowBuild), a minimum release age, and a trust policy that gates which packages may run lifecycle scripts. Three updates required changes beyond the version numbers.

The clearest case was minimatch 3.1.5, which several tools pull in transitively. A transitive brace-expansion 5.x release changed its export shape from a callable default to a named { expand }, and minimatch 3.1.5 called it as a function, so it broke. The fix was a one-line pnpm patch that supports both shapes:

// patches/minimatch@3.1.5.patch
-var expand = require('brace-expansion')
+var __braceExpansion = require('brace-expansion')
+var expand = typeof __braceExpansion === 'function'
+  ? __braceExpansion
+  : (__braceExpansion.expand || __braceExpansion.default || __braceExpansion)

That patch sits in patches/ and applies automatically on install, so every consumer of the old minimatch gets the fix without forking or waiting on an upstream release.

Two more supply-chain moves. Scoped overrides reached transitive vulnerabilities that a direct bump could not, because the vuln lived in a sub-dependency no manifest directly referenced. And @react-native-firebase 24.1.1 switched to provenance attestation, which the repo’s trust policy did not yet accept, so it landed in trustPolicyExclude as a deliberate downgrade rather than weakening the policy globally.

A version bump, a breaking export change, an unreachable transitive vulnerability, and a new provenance requirement each required a different mitigation: a patch, an override, or a trust exclusion. I classified each update before applying the change.

Terraform CI from zero

Terraform had no CI coverage at all. Provider bumps merged on eyeballing the diff. I added a credential-free gate that runs across all sixteen directories (the ten stack roots plus six modules) without touching the cloud:

TierWhat it checksCredential?Blocks merge?
1fmt, init -backend=false, validate, lockfile-sync, required_version consistencynoneyes
2plan drift vs remote statecloud (manual)no
3promote to prodhumanno

Tier 1 runs on every MR. Tier 2 is a manual terraform plan against remote state, and Tier 3 is the human promotion gate. The credential-free gate blocks changes that fail its checks, while the credential-bearing plan remains manual.

Several implementation details became apparent while standing this up.

Portable lockfiles need both platforms. Terraform writes .terraform.lock.hcl with hashes for the platform it ran on. CI runs on Linux; developers run on macOS arm64. A lock committed from one fails on the other. Every lockfile has to carry both:

terraform providers lock -platform=linux_amd64 -platform=darwin_arm64

init is not enough after a bump; you need -upgrade. A plain terraform init refuses to install a provider when the committed lock does not satisfy a newly-bumped constraint. The lockfile-sync check then fails because init is honoring the old lock. The fix is terraform init -upgrade, which re-resolves, followed by providers lock to make the result portable. I wrote a regen_terraform_locks.mjs script for this and then had to fix it in a follow-up, because its first version did not pass -upgrade and so could not handle its own primary use case.

required_version should be a range, not a pin. The stacks had required_version = "= 1.15.x" style pins. That couples every stack to the exact CLI version and makes Renovate’s terraform-manager bump noisy. Changing to required_version = "~> 1.15" lets the CLI move within the minor without touching every stack.

Local environment can change credential-free checks. infrastructure/mise.toml exports AWS_PROFILE=terraform for local use. A local check:terraform therefore uses that profile, while CI runs without it. To match CI locally, I used env -u AWS_PROFILE before running the gate.

Task names collide. There were two terraform tasks: //:check:terraform (the root task, sixteen directories, the one CI uses) and //infrastructure:terraform:check (a legacy four-stack task, local only). Running the wrong one gives a pass that does not mean what you think it means. The root task represents the full check; the legacy task covers only four stacks.

The AWS provider update (6.41.0 to 6.56.0 across ten stacks, plus the Lambda module 8.7.0 to 8.8.0) verified that the gate worked. It regenerated every lock with -upgrade plus providers lock, and the Tier-1 gate validated all sixteen directories in CI before merge.

The CI gate audit: where CI did not match check:all

The next issue came from a regression. A vue-tsc dependency bump shipped a typecheck error into web-app. It passed locally and passed CI, because web-app had no typecheck job in its MR pipeline at all. Only web-app:build:artifact ran on .vue changes, and a Vite production build (esbuild) does not typecheck.

That sent me auditing which projects actually enforced their full local gate in CI. The :quality jobs are the gate that runs typecheck plus lint plus test. The picture was uneven:

Project:quality in CI before?Enforced typecheck?
frontend-nuxtyesyes
cms-appyesyes
mobile-appyesyes
e2e-testsyesyes
web-appnono
admin-nuxtnono
backendpartial (test + build only)no static analysis

Three projects lacked equivalent CI gates. web-app and admin-nuxt had no :quality job at all. The backend ran tests and a production build in CI, but the backend quality checks (Spotless, Checkstyle, PMD, and SpotBugs) were local-only. The regression resulted from the CI job configuration.

I added all three. Each one broke in CI in a different way, which is the rest of this post.

Another CI-shape behavior came from a changes: rule that lists source paths. If an MR touches only paths matched by no job (here, docs/*.md and scripts/*.mjs), the MR receives no pipeline. With “pipelines must succeed” enforced, that MR cannot merge because no pipeline exists. I added a git-diff-scoped docs:lint job. changes: filters therefore determine both which jobs run and whether the MR receives a pipeline.

Frontend typecheck failure modes

Adding the two frontend :quality jobs exposed three different CI and local-environment differences.

The inline env prefix only applies to the first command

admin-nuxt’s typecheck passed locally and OOM’d in CI:

"typecheck": "NODE_OPTIONS='--max-old-space-size=4096' nuxt prepare && vue-tsc --noEmit"

In a shell, the inline VAR=val prefix applies only to the first command in the && chain. vue-tsc ran at the default ~2 GB heap and exited. The 4096 MB setting reached only nuxt prepare. The fix is to apply the prefix to the command that uses the memory, or export it for the line:

"typecheck": "nuxt prepare && NODE_OPTIONS='--max-old-space-size=4096' vue-tsc --noEmit"

I first changed the heap size, but the failure remained because the NODE_OPTIONS prefix applied only to nuxt prepare. The prefix had to be applied to vue-tsc, which was the process using the memory.

Generated artifacts and fresh checkouts

web-app’s typecheck depends on src/auto-imports.d.ts, which unplugin-auto-import emits only during a vite build. The file is gitignored. It exists on a developer’s machine because some earlier build left it there, so local typecheck passes. On a fresh CI checkout it does not exist, and typecheck fails with Cannot find name 'ref' for every auto-imported symbol.

The :quality job now runs build before typecheck to regenerate it. A typecheck that depends on a build artifact can pass locally when the artifact exists and fail on a clean CI tree.

Non-primitive defineModel defaults

The dependency bump exposed a Vue reactivity issue that a stricter type-checker surfaced. ImageDropFiles.vue had:

defineModel<Item[]>("items", { default: [] });

{ default: [] } shares a single array literal across component instances. Under stricter vue-tsc settings that fails typecheck. The factory form supplies a separate default for each component instance:

defineModel<Item[]>("items", { default: () => [] });

The bug was present before the vue-tsc upgrade. The upgrade exposed it without introducing the incorrect default.

Versions that cannot auto-bump yet

A few dependencies are pinned behind known breakage that a version bump alone cannot fix. They are migration tasks, not Renovate tasks:

DepPinned atWhy it cannot move
orval8.2.0 / 8.5.38.22.0 generates TypeScript incompatible with vue-tsc, and drifts the API contracts
msw2.12.72.12.14+ locks global.fetch as read-only, breaking test setup
vue-tsc3.2.x (nuxt apps)3.3.x tightens inline @click handler typing, producing ~229 errors in two frontends

Until those land, the corresponding majors stay closed. web-app is the exception: it can take vue-tsc 3.3.7, but only with the defineModel factory-default fix above applied.

Backend static analysis cascade

The backend’s :quality job runs the static-analysis checks that used to be local-only: Spotless, Checkstyle, PMD, and SpotBugs. Turning it on in CI surfaced a batch of PMD 7.26 violations, and fixing them cascaded.

UseUtilityClass says a class with only static members should not be instantiable. One possible fix is to add a private constructor. That then triggers ClassWithOnlyPrivateConstructorsShouldBeFinal, which requires the class to be final. A final class with only private constructors then triggers MissingStaticMethodInNonInstantiatableClass. I applied final and a private constructor to constants holders. I left marker classes such as Views, which exists to carry Jackson annotations, with @SuppressWarnings because changing their instantiation model would not affect the annotation use.

VariableDeclarationUsageDistance arrived with PMD 7.25.0 and measures how far a variable declaration sits from its first use. The property that tunes it is maxDistance, not distance. My first configuration used distance and had no effect. The default value of 7 produced violations in this codebase, so I set maxDistance to 25 for this readability rule:

<!-- backend/pmd-ruleset.xml -->
<rule ref="category/java/codestyle.xml/VariableDeclarationUsageDistance">
  <properties>
    <property name="maxDistance" value="25"/>
  </properties>
</rule>

A linter bump can add rules, change rule behavior, and expose interactions between rules. I raised the threshold for this readability rule and kept correctness changes in production code.

Pipeline and runner mechanics

A few lessons do not fit a single category. They are about GitLab pipelines as infrastructure.

Detached pipelines. To avoid re-running the full set, I triggered a test pipeline through the API (POST /projects/:id/pipeline?ref=<branch>). It passed, but the merge request stayed ci_must_pass. A pipeline started that way is detached: it is not associated with the MR, so it does not satisfy the MR’s pipeline gate even though it ran the same jobs on the same commit. Retrying the MR’s own push-created pipeline (POST /pipelines/:id/retry) reruns it while keeping the MR association. Only pipelines GitLab creates in the MR’s context count toward the merge gate.

Transient failures. GitLab jobs intermittently fail with HTTP Basic: Access denied, a transient auth blip that clears on retry. I distinguished that from an underlying job failure before debugging further. The trace showed a transient auth error; retrying the job succeeded.

Runner scarcity. With two runners, an unchecked main pipeline delays subsequent MR validation. Canceling the main pipeline after each merge kept the validation queue moving. On a few occasions, a “stuck” job was waiting for a runner that another job had not released yet.

Canceling a pipeline changes the state of its jobs. When I canceled a running pipeline and then retried a single job from it, the retry left the pipeline with jobs in created and the pipeline in canceled. Retrying the whole pipeline or pushing an empty commit created a new linked pipeline. A single-job retry did not reset the canceled pipeline.

The majors, routed through the dashboard

The major upgrades were routed through the Dependency Dashboard. I closed them individually according to the project’s stated policy, leaving them available for explicit approval after the migration work for each major is understood.

The flip side is that those majors do not disappear. They sit on the dashboard waiting for the migrations that unblock them (the orval, msw, and vue-tsc work above). Closing them was a policy decision, not a resolution.

Where this leaves us

After all of it, check:all runs fully green on main, and every gate check:all runs locally now also runs in CI. Before this pass, check:all and CI differed in three places (web-app typecheck, admin-nuxt typecheck, and backend static analysis). The local result therefore did not guarantee the same CI checks. The configurations now include the same gates.

What remains to reach hands-off dependency automation:

  • Unblock the pinned versions. The msw and vue-tsc migrations remain. Once they land, the closed majors can be reconsidered and the pins can be removed.
  • Make the gate agreement a contract. The audit that found three missing :quality jobs was manual. A check that fails when a project’s local gate is not represented in CI would stop the next drift from going unnoticed.
  • Configure automerge for patch and minor lanes. Once every MR’s CI represents the local gates, automerge: true with automergeType: pr can be considered for grouped patch and minor lanes. Major upgrades and the pinned migrations still require review.

The observed failure points were stale branches, lockfiles, linter cascades, generated artifacts, changes-filter gaps, and gates that existed locally but were absent from the pipeline. Renovate opened the MRs; the remaining work was to align the CI checks with the local checks and handle migrations that Renovate could not perform automatically.

References

This post was written with AI assistance.