fix(docker): clone ffmpeg from the github mirror and pin to n8.1.2 - #4025
Merged
Conversation
git.ffmpeg.org intermittently returns HTTP 502 on clone, which fails the builder image with exit 128. It has failed from both GitHub hosted and self hosted runners while succeeding from a local machine minutes later. The cause on the server side is unconfirmed. All ffmpeg.org hostnames resolve to a single address with no failover, which is consistent with a proxy returning 502 under clone load, but we cannot verify that. github.com/FFmpeg/FFmpeg is not an official FFmpeg endorsed mirror; it self describes as a mirror and is current. This build already depends on github.com for grpc_health_probe and tasmodel.pb. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The FFMPEG_SHA pin never took effect. VAR=value command is a POSIX prefix assignment, so the variable only reached git clone's environment; both later commands expanded it to an empty string, leaving git fetch on the default branch and bare git checkout a silent no-op. Every builder image has compiled upstream master as of build time. git clone accepts tags but not SHAs, which is why the original needed clone-then-fetch. Pinning a tag removes the variable entirely. n8.1.2 over n9.0.1 for maturity: eight weeks in the field against a tag cut hours ago. Verified to configure and build with these flags. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This was referenced Aug 12, 2026
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #4025 +/- ##
===================================================
- Coverage 35.54450% 35.54008% -0.00442%
===================================================
Files 174 174
Lines 45225 45225
===================================================
- Hits 16075 16073 -2
- Misses 27877 27879 +2
Partials 1273 1273 see 2 files with indirect coverage changes Continue to review full report in Codecov by Harness.
🚀 New features to boost your workflow:
|
j0sh
approved these changes
Aug 14, 2026
Collaborator
|
Merging because I keep hitting this, thanks |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Combines #4024 and #4022 into one PR, as requested in review. Both were approved separately. Kept as two commits so each half stays reviewable and revertable.
1. Clone from the GitHub mirror (#4024)
git.ffmpeg.orgintermittently returns HTTP 502, failing the clone withexit code: 128:It broke the v0.9.1 build three times from GitHub hosted runners and once from a self hosted runner, while cloning fine from a local workstation in between. Details in #4023.
The server side cause is unconfirmed and
github.com/FFmpeg/FFmpegis not an officially endorsed mirror; see #4023 and #4024 for the full caveats. Note the build already clones fromgithub.comthree times, including twice frominstall_ffmpeg.shvia lpms, so this does not add a dependency that is not already load bearing.2. Pin ffmpeg to n8.1.2 (#4022)
The
FFMPEG_SHApin never took effect.VAR=value commandis a POSIX prefix assignment, so the variable only enteredgit clone's environment. Both later commands expanded it to an empty string:git fetchtook the default branch, and baregit checkoutexited 0. That silent success is why it went unnoticed since #3353 (Feb 2025), during which every builder image compiled upstreammasteras of build time.git cloneaccepts tags but not commit SHAs, which is why the original needed the clone-then-fetch dance. Pinning a tag removes the variable, and therefore the whole bug class.n8.1.2overn9.0.1for maturity: eight weeks in the field versus a tag cut hours before. All ofn9.0.1,n9.0andn8.1.2were verified to configure and build in a replica of the builder environment (ubuntu 20.04, libx264 0.155, libfdk-aac 0.1.6, gcc 9, nasm 3.01) with the exact flags from this file.n8.1.2contains the HEVC-in-FLV commit the original pin was chosen for.Verification
The builder job skips its build step on PRs (gated on
github.ref_namematchingmain|master|vX.Y.Z), so a green check here does not exercise this layer. On #4024 the guard was temporarily disabled to test it for real, then removed:https://github.com/livepeer/go-livepeer/actions/runs/31608535985
go-livepeer builder docker image generationubuntu-24.04Docker image generationThat run cold built the ffmpeg layer from the mirror on a hosted runner, which is where the 502s occurred. It did not include the
n8.1.2pin, which was verified locally instead.Behaviour change
Because the pin never worked, images since Feb 2025 shipped whatever
masterwas at build time. This is the first build to honour a pin, so the shippedffmpeg/ffprobeand the libs under/usr/local/libmove ton8.1.2. The CGO link target is unaffected: that comes frominstall_ffmpeg.shvia lpms and is pinned separately.Supersedes #4022 and #4024. Closes #4021. Closes #4023.