Build cache ERROR: failed to solve: Internal: not found

[bhperry]

Contributing guidelines and issue reporting guide

• I've read the contributing guidelines and wholeheartedly agree. I've also read the issue reporting guide.

Well-formed report checklist

• I have found a bug that the documentation does not mention anything about my problem
• I have found a bug that there are no open or closed issues that are related to my problem
• I have provided version/information about my environment and done my best to provide a reproducer

Description of bug

Bug description

Lately I have started to see docker builds in Github Actions that use the GHA build cache fail regularly when exporting the cache at the end of a build. Re-running the build will often cause it to succeed, but we get the failures almost every day now.

Have been running with the GHA cache for a while now with no issues.

#9 exporting to GitHub Actions Cache
#9 preparing build cache for export
#9 preparing build cache for export 0.4s done
#9 DONE 0.4s
ERROR: failed to solve: Internal: not found

Reproduction

Happens with a variety of different image builds, some of which are very thin wrappers on existing images. So any basic Dockerfile should work.

We run the builds from within a github action, automated by python scripts. But the end result is to run commands like this:

docker buildx build --cache-from type=gha,scope=build-cache-daily-${NAME} --cache-to type=gha,scope=build-cache-daily-${NAME},mode=max,ghtoken=${GH_TOKEN},repository=${REPO}  --tag test:latest -f Dockerfile .

Version information

ubuntu-20.04 github actions runner

Docker-Buildx 0.21.1
Docker Client 26.1.3

[tonistiigi]

Might be related to Github moving to V2 version of their API https://github.blog/changelog/2024-09-16-notice-of-upcoming-deprecations-and-changes-in-github-actions-services/

[bhperry]

I did see some other mentions of upcoming cache changes. The fact that it generally does not happen on retry made me think it wasn't that. The brownout dates are certainly interesting, but it happened this morning which is not in the listed dates.

[tonistiigi]

We already have v2 implementation in BuildKit v0.20.0 . Afaics, most repositories are already using v2 when runners start, although at least initially only some repositories had support. The deadlines are for shutting down v1.

[tonistiigi]

If you have a link for your Github actions run then you can post it for more details.

[bhperry]

It says deprecation of v1-v2 in that article, which was my previous understanding as well. Only v3-v4 supported in future. Ah I understnd now, V2 of the API does not map to the versioning of the action.

This is on a private repository unfortunately, can't link.

[Link-]

@bhperry - We'd be very grateful if you would reach out to the GitHub support team, open a ticket and drop the link to your run in the private repository. We'd like to take a look to understand this behaviour. Feel free to mention this comment as well and ask the team to escalate to engineering.

[bhperry]

@Link- ticket submitted

[crazy-max]

We run the builds from within a github action, automated by python scripts. But the end result is to run commands like this:

docker buildx build --cache-from type=gha,scope=build-cache-daily-${NAME} --cache-to type=gha,scope=build-cache-daily-${NAME},mode=max,ghtoken=${GH_TOKEN},repository=${REPO} --tag test:latest -f Dockerfile .

@bhperry Does this command runs within a run block? If so ACTIONS_* envs are not exposed. But I guess as you see exporting to GitHub Actions Cache in the logs they get exposed.

Also does it change smth if you remove these attributes: ,ghtoken=${GH_TOKEN},repository=${REPO} ?

Can you show your workflow as well please?

And what is the output of docker buildx ls?

[bhperry]

@crazy-max Yes it does run within a run block. I have a local action exporting ACTIONS_CACHE_URL and ACTIONS_RUNTIME_TOKEN, similar to your ghaction-github-runtime action.

I'm going to try to make a simpler reproduction of this, because there's a lot of scripts internal to my organization obfuscating what's actually happening in the workflow.