In the world of DevOps and CI/CD pipelines, GitHub Actions stands out as a
powerful tool for automation. However, as with all tools, the devil lies in the
details; or in this case, the dependency chain. This post dives into the recent
horror story we faced when we realized that our armv7l-based self-hosted
runners cannot connect to Github in Ubuntu 24.04.
The Root of the Problem: Y2038 and .NET Compatibility
Link to heading
Ubuntu 24.04 introduced a critical fix to the Y2038 problem for armv7l. By
shifting to 64-bit time values, it rendered many older libraries and binaries
incompatible. One such casualty was the GitHub Actions runner.
GitHub’s runner for armv7l is built using .NET 8.0, which remains
incompatible with the Y2038 fix. This manifests as SSL connection failures
with errors like:
1The remote certificate is invalid because of errors in the certificate chain: NotTimeValid
The problem stems from outdated libraries in .NET 8.0 that don’t handle the
updated time format correctly. While .NET 9.0 has introduced Y2038
compatibility, backports to .NET 8.0 are not planned, as confirmed in the
dotnet/core discussions.
DIY Solution: Rebuilding the Runner Link to heading
The immediate workaround involves rebuilding the GitHub Actions runner with
.NET 9.0. However, this is easier said than done. The official documentation
lacks guidance for such rebuilds, and even attempts with .NET 9.0 SDKs often
result in the same issues persisting.
Fortunately, the community stepped in; satmandu
provided a pre-built binary release compatible with Ubuntu 24.04, which can be
found here. Testing
confirmed that this binary works where manually rebuilt runners failed.
The Broader Implications Link to heading
This issue highlights the challenges of operating in a diverse ecosystem.
Moving to Y2038-compatible time values was a necessary but disruptive step,
breaking backward compatibility in critical infrastructure. The reliance on
older .NET versions by GitHub Actions created a brittle link in the chain.
Luckily, the open-source community played a crucial role in bridging the gap
with patches and pre-built binaries.
If you’re managing self-hosted runners, test against the latest operating
systems and library versions regularly. Be aware of the technology stack
underpinning critical systems. As demonstrated here, the interaction between
.NET and Ubuntu was at the heart of the issue. Follow GitHub and .NET
release notes closely. Breaking changes like those in .NET 9.0 can have
far-reaching effects.
The armv7l runner saga is a cautionary tale for the DevOps community. While
the path forward involves embracing .NET 9.0 and its support for Ubuntu
24.04, the transition highlights the complexities of maintaining compatibility
across evolving platforms.
For now, if you’re running Ubuntu 24.04 on armv7l, grab the patched
binary or dive into
the rebuilding process - just be prepared for some manual effort.
If you’ve faced similar challenges or have alternative solutions, feel free to share and drop us a line!