Skip to content

29.7.0 (containerd image store) breaks docker push with "blob unknown to registry"; references blob skipped as "already exists" #53293

Description

@mtslzr

Description

Since upgrading our GitLab CI dind service from docker:29.6.2-dind to docker:29.7.0-dind (both run with --feature=containerd-snapshotter=true), every docker push to our private GitLab registry fails at the final upload with:

error from registry: blob unknown to registry - sha256:….

All layers report Pushed or Layer already exists, but the digest the registry rejects never appears in the upload progress; the daemon appears to skip uploading a blob it wrongly believes already exists in the target repository. It appears to be tied to the 29.7 release; all CI jobs with image uploaded started failing after we upgraded dind; and they all worked again when reverted to 29.6.2.

The affected images share common base layers (node:24.18.1-slim) that exist in sibling repositories under the same registry namespace, so cross-repo layer dedup/mount bookkeeping looks implicated; the only push-path change we can find between the releases is #53081, which newly threads the upload semaphore into findMissingMountable and the push dispatch.

29.7.1 did not appear to fix it; let me know if I can provide more information.

Reproduce

  1. Start a Docker 29.7.0 daemon with the containerd image store enabled (in our case: docker:29.7.0-dind image run via GitLab CI, with --feature=containerd-snapshotter=true).
  2. Authenticate to a private registry that already contains sibling repositories sharing base layers with the image you're about to push.
  3. docker build an image on top of one of those shared base images.
  4. docker push <registry>/<repo>:<tag>, and see the upload fail, as described above.

Expected behavior

Push completes, upload is successful.

docker version

Client:
 Version:           29.7.0
 API version:       1.55
 Go version:        go1.26.5
 Git commit:        c1eba93
 Built:             Thu Jul 30 20:19:48 2026
 OS/Arch:           linux/amd64
 Context:           default
Server: Docker Engine - Community
 Engine:
  Version:          29.7.0
  API version:      1.55 (minimum version 1.40)
  Go version:       go1.26.5
  Git commit:       4b5cb71
  Built:            Thu Jul 30 20:22:11 2026
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v2.3.3
  GitCommit:        aad11006b869517fcd3009450b6f82da282e1a9b
 runc:
  Version:          1.4.3
  GitCommit:        v1.4.3-0-gbb14dab
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Server:
 Server Version: 29.7.0
 Storage Driver: overlayfs
  driver-type: io.containerd.snapshotter.v1
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 containerd version: aad11006b869517fcd3009450b6f82da282e1a9b
 runc version: v1.4.3-0-gbb14dab
 Operating System: Alpine Linux v3.24 (containerized)
 OSType: linux
 Architecture: x86_64
 Registry Mirrors:
  https://mirror.gcr.io/
Client:
 Plugins:
  buildx: v0.36.0
  compose: v5.3.1

Additional Info

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/distributionImage Distributionkind/bugBugs are bugs. The cause may or may not be known at triage time so debugging may be needed.version/29.7

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions