Docker gradually exhausts disk space on BTRFS #27653

Description

Steps to reproduce the issue:

1. Install docker on a btrfs system
2. Use docker intensively for a while, and make sure to remove & recreate containers, rebuild with no cache, restart the system, ...
3. Check disk space

Describe the results you received:

• /var/lib/docker uses up more than 30GB, with all except a few megabytes located inside /var/lib/docker/btrfs

• docker ps -s shows a few gigabtes, below 10GB

• The following do absolutely nothing to change the /var/lib/docker disk space use

  1. docker rmi $(docker images -aq)
  2. docker volume rm on every line of docker volume ls -qf dangling=true
  3. removing all files matching /var/lib/docker/*/*-json.log
  4. docker rm on every stopped container listed in docker ps -a

  If there is an obvious cleanup procedure that I missed, it's probably not a bug but my own mistake.
  I have been known to wonder why disk space runs out so quickly with docker logs growing to gigabytes, without realizing what was going on 😄

• The following solves the situation (but it shouldn't be required):

  1. <shutdown all services>
  2. service docker stop
  3. apt-get remove --purge docker-engine
  4. rm -rf /var/lib/docker/
  5. apt-get install -y docker-engine
  6. service docker start
  7. <restart all services>

  After this procedure, disk usage of /var/lib/docker goes back to a few single digit gigabytes as expected

Describe the results you expected:
/var/lib/docker is, after the failing cleanup procedures described above, not significantly larger than what is shown in docker ps -s

Additional information you deem important (e.g. issue happens only occasionally):

Output of docker version:

root@Ubuntu-1510-wily-64-minimal ~ # docker version
Client:
 Version:      1.12.2
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   bb80604
 Built:        Tue Oct 11 18:29:41 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.2
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   bb80604
 Built:        Tue Oct 11 18:29:41 2016
 OS/Arch:      linux/amd64
root@Ubuntu-1510-wily-64-minimal ~ #

Output of docker info:
(this is after purging and reinstalling as described above)

root@Ubuntu-1510-wily-64-minimal ~ # docker info
Containers: 14
 Running: 14
 Paused: 0
 Stopped: 0
Images: 41
Server Version: 1.12.2
Storage Driver: btrfs
 Build Version: Btrfs v4.4
 Library Version: 101
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: null host bridge overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: apparmor seccomp
Kernel Version: 4.4.0-45-generic
Operating System: Ubuntu 16.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.859 GiB
Name: Ubuntu-1510-wily-64-minimal
ID: 2ILP:SBMD:2NB7:JY7W:YC3K:CYYV:AX5E:FZLB:CON4:3JCT:3GT4:W4PB
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Insecure Registries:
 127.0.0.0/8
root@Ubuntu-1510-wily-64-minimal ~ #

Additional environment details (AWS, VirtualBox, physical, etc.):
KVM virtual server hosted by hetzner, operating system is Ubuntu 16.04 LTS (the 15.10 occurances above are simply unchanged terminal prompts, the system has been upgraded a while ago)