Description
Since upgrading our GitLab CI dind service from docker:29.6.2-dind to docker:29.7.0-dind (both run with --feature=containerd-snapshotter=true), every docker push to our private GitLab registry fails at the final upload with:
error from registry: blob unknown to registry - sha256:….
All layers report Pushed or Layer already exists, but the digest the registry rejects never appears in the upload progress; the daemon appears to skip uploading a blob it wrongly believes already exists in the target repository. It appears to be tied to the 29.7 release; all CI jobs with image uploaded started failing after we upgraded dind; and they all worked again when reverted to 29.6.2.
The affected images share common base layers (node:24.18.1-slim) that exist in sibling repositories under the same registry namespace, so cross-repo layer dedup/mount bookkeeping looks implicated; the only push-path change we can find between the releases is #53081, which newly threads the upload semaphore into findMissingMountable and the push dispatch.
29.7.1 did not appear to fix it; let me know if I can provide more information.
Reproduce
- Start a Docker 29.7.0 daemon with the containerd image store enabled (in our case:
docker:29.7.0-dind image run via GitLab CI, with --feature=containerd-snapshotter=true).
- Authenticate to a private registry that already contains sibling repositories sharing base layers with the image you're about to push.
docker build an image on top of one of those shared base images.
docker push <registry>/<repo>:<tag>, and see the upload fail, as described above.
Expected behavior
Push completes, upload is successful.
docker version
Client:
Version: 29.7.0
API version: 1.55
Go version: go1.26.5
Git commit: c1eba93
Built: Thu Jul 30 20:19:48 2026
OS/Arch: linux/amd64
Context: default
Server: Docker Engine - Community
Engine:
Version: 29.7.0
API version: 1.55 (minimum version 1.40)
Go version: go1.26.5
Git commit: 4b5cb71
Built: Thu Jul 30 20:22:11 2026
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: v2.3.3
GitCommit: aad11006b869517fcd3009450b6f82da282e1a9b
runc:
Version: 1.4.3
GitCommit: v1.4.3-0-gbb14dab
docker-init:
Version: 0.19.0
GitCommit: de40ad0
docker info
Server:
Server Version: 29.7.0
Storage Driver: overlayfs
driver-type: io.containerd.snapshotter.v1
Cgroup Driver: cgroupfs
Cgroup Version: 2
containerd version: aad11006b869517fcd3009450b6f82da282e1a9b
runc version: v1.4.3-0-gbb14dab
Operating System: Alpine Linux v3.24 (containerized)
OSType: linux
Architecture: x86_64
Registry Mirrors:
https://mirror.gcr.io/
Client:
Plugins:
buildx: v0.36.0
compose: v5.3.1
Additional Info
No response
Description
Since upgrading our GitLab CI dind service from
docker:29.6.2-dindtodocker:29.7.0-dind(both run with--feature=containerd-snapshotter=true), every docker push to our private GitLab registry fails at the final upload with:All layers report Pushed or Layer already exists, but the digest the registry rejects never appears in the upload progress; the daemon appears to skip uploading a blob it wrongly believes already exists in the target repository. It appears to be tied to the 29.7 release; all CI jobs with image uploaded started failing after we upgraded dind; and they all worked again when reverted to 29.6.2.
The affected images share common base layers (node:24.18.1-slim) that exist in sibling repositories under the same registry namespace, so cross-repo layer dedup/mount bookkeeping looks implicated; the only push-path change we can find between the releases is #53081, which newly threads the upload semaphore into
findMissingMountableand the push dispatch.29.7.1 did not appear to fix it; let me know if I can provide more information.
Reproduce
docker:29.7.0-dindimage run via GitLab CI, with--feature=containerd-snapshotter=true).docker buildan image on top of one of those shared base images.docker push <registry>/<repo>:<tag>, and see the upload fail, as described above.Expected behavior
Push completes, upload is successful.
docker version
Client: Version: 29.7.0 API version: 1.55 Go version: go1.26.5 Git commit: c1eba93 Built: Thu Jul 30 20:19:48 2026 OS/Arch: linux/amd64 Context: default Server: Docker Engine - Community Engine: Version: 29.7.0 API version: 1.55 (minimum version 1.40) Go version: go1.26.5 Git commit: 4b5cb71 Built: Thu Jul 30 20:22:11 2026 OS/Arch: linux/amd64 Experimental: false containerd: Version: v2.3.3 GitCommit: aad11006b869517fcd3009450b6f82da282e1a9b runc: Version: 1.4.3 GitCommit: v1.4.3-0-gbb14dab docker-init: Version: 0.19.0 GitCommit: de40ad0docker info
Additional Info
No response