When a Release Breaks Your CI Link to heading

We’re a small engineering team. Everyone’s busy! Some days we’re deep in container runtime dev, other days we’re debugging transport layers for vAccel or measuring latency for torch model execution offloading across Edge devices. What we don’t have is a dedicated team for CI maintenance.

So when our GitHub Actions runners went down again, we heard the familiar chorus:

“CI is down.”
“Is the cluster up?”
“Did someone change something?”

Nobody had. The culprit was subtler and (once more) frustrating.

GitHub follows a 30-day policy to update the runner software. Specifically:

Any updates released for the software, including major, minor, or patch releases, are considered as an available update. If you do not perform a software update within 30 days, the GitHub Actions service will not queue jobs to your runner. In addition, if a critical security update is required, the GitHub Actions service will not queue jobs to your runner until it has been updated.

But if you’re like us, running self-hosted GH runners on containers, then the auto-update feature of the runner does not scale well.

As a result, when v2.328.0 was released, the older v2.327.1 became unsupported. Our self-hosted runners, built with the old version, were simply not receiving jobs by GitHub.

Just a broken CI.

We were wasting hours each time this happened: manually updating runner versions, rebuilding images, and redeploying. We needed a fix that was automatic, self-contained, and didn’t require human babysitting.

The Key Insight: Use Dependabot as a Build Trigger Link to heading

Dependabot is best known for keeping dependencies up to date. It can bump versions of Go packages, rust crates, container images, and even GitHub Actions.

That last part, GitHub Actions, turned out to be our way out.

What if we asked Dependabot to track the version of actions/runner,
and used its pull request as the trigger to rebuild our runner images?

That single insight became the foundation of a hands-free CI maintenance pipeline.

We will follow-up on another post about our CI setup, as we have upgraded to ARC since our last post about it. Runner images are mostly based on some-natalie’s kubernoodles.

Step 1: The Tracker Link to heading

We added a minimal file to .github/workflows:

1# _track_runner.yml
2uses: actions/runner@v2.329.0

Then we configured Dependabot to watch that folder:

1# .github/dependabot.yml
2version: 2
3updates:
4  - package-ecosystem: "github-actions"
5    directory: "/.github/workflows"
6    schedule:
7      interval: "daily"

That’s it.

Whenever GitHub publishes a new actions/runner release, Dependabot opens a PR like:

chore(deps): Bump actions/runner from v2.327.1 to v2.329.0

That PR became our signal: the moment Dependabot does its job, we rebuild our runners automatically.

Step 2: The Trigger Workflow Link to heading

When Dependabot’s PR gets issued, it triggers a lightweight workflow that extracts the runner version and dispatches a full image rebuild.

 1# .github/workflows/build-trigger.yml
 2on:
 3  pull_request:
 4    branches: [ main ]
 5    paths:
 6      - ".github/workflows/_track_runner.yml"
 7
 8jobs:
 9  extract_version:
10    runs-on: ubuntu-latest
11    outputs:
12      runner-version: ${{ steps.extract.outputs.version }}
13    steps:
14      - uses: actions/checkout@v5
15        with: 
16          ref: ${{ github.event.pull_request.head.ref }}
17          fetch-depth: 0
18
19      - id: extract
20        run: |
21          version=$(grep -oE 'actions/runner@v[0-9]+\.[0-9]+\.[0-9]+' .github/workflows/_track_runner.yml | cut -d@ -f2 | sed 's/^v//')
22          echo "version=$version" >> $GITHUB_OUTPUT          
23
24  build_base:
25    needs: extract_version
26    uses: ./.github/workflows/build-latest.yml
27    with:
28      runner-version: ${{ needs.extract_version.outputs.runner-version }}
29      runner: '["base","dind","2204"]'
30      runner-archs: '["amd64","arm64","arm"]'
31      dockerfiles: '["jammy-base","noble-base"]'
32    secrets: inherit
33
34  build_custom:
35    needs: [extract_version,build_base]
36    uses: ./.github/workflows/build-latest.yml
37    with:
38      runner-version: ${{ needs.extract_version.outputs.runner-version }}
39      runner: '["base","dind","2204"]'
40      runner-archs: '["amd64","arm64"]'
41      dockerfiles: '["jammy-tf","jammy-torch","jammy-tvm","jammy-opencv"]'
42    secrets: inherit
43
44  auto_merge:
45    needs: [build_base,build_custom]
46    if: github.actor == 'dependabot[bot]'
47    runs-on: ubuntu-latest
48    steps:
49      - name: Auto-merge Dependabot PR
50        run: |
51          gh pr merge ${{ github.event.pull_request.number }} --rebase --delete-branch --admin --repo ${{ github.repository }}          
52        env:
53          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

That small workflow is the bridge: it detects the change, parses the version, and tells our builder to get to work.

Step 3: Building & Signing Custom Runner Images Link to heading

Our main workflow, build-latest.yml, handles the heavy lifting:

  • Multi-arch container image builds for amd64, arm64, and arm
  • Image publishing to our Harbor registry
  • cosign signing for supply chain integrity

Example output images:

1harbor.nbfc.io/nubificus/runner-images/jammy-base:amd64-d3aa6e9
2harbor.nbfc.io/nubificus/runner-images/jammy-base:arm64-d3aa6e9

We sign these images using cosign in keyless mode (using GH’s OIDC) and we merge the per-arch images into a manifest which we also sign:

Example signing process:

1cosign sign --yes ${{ env.REGISTRY }}/${{ github.repository }}/${{ matrix.dockerfile }}@$DIGEST \
2  -a "repo=${{github.repository}}" \
3  -a "workflow=${{github.workflow}}" \
4  -a "runner_version=${{ inputs.runner-version }}" \
5  -a "author=Nubificus LTD"

We use Harbor robot accounts for pushing images. Since Harbor doesn’t yet support cosign’s latest bundle spec update (v3.x), we use the referrers API but keep the old format for the signature.

Step 4: Runners That Update Themselves Link to heading

Now, whenever GitHub releases a new runner:

  • Dependabot notices the new version and opens a PR
  • Our trigger workflow fires
  • The build workflow rebuilds and publishes new images
  • Once the workflow succeeds, it is being auto-merged.
  • Our runner infrastructure updates itself automatically, since the imagePullPolicy is set to Always

No more manual updates. No more “why is CI down again?” messages. No one needs to touch anything.

The Impact for a Small Team Link to heading

For a small, distributed team like ours, this bit of automation made a huge difference:

  • Zero manual maintenance: runners rebuild automatically
  • No more noise: engineers stopped pinging “CI down again?” in chat
  • Secure by default: each build is signed and traceable
  • Always up to date: Dependabot ensures we track upstream releases within hours

We now have a fully self-maintaining CI backbone, built with the same tools we already use, no new services, no new costs, and no dedicated CI admin.

Epilogue: The Dependabot Hack Link to heading

It’s funny ;) Dependabot wasn’t built for this. It’s a dependency updater, not a workflow orchestrator. But it’s also the perfect sensor for upstream version changes.

By wiring it into our build system, we turned it into the simplest and most reliable CI babysitter we could have asked for. Sometimes, the most powerful DevOps automation comes from re-purposing what’s already there.