Docker Issues and Tips (aufs/overlay/btrfs..)
October 10, 2017 · View on GitHub
Picked up and categorized subjectively from https://github.com/docker/docker/issues. Comments and pull requests are welcome.
:white_large_square: = Open (maybe not up-to-date, please check the link by yourself!)
:white_square_button: = Mostly resolved (ditto, plus subjective)
:white_check_mark: = Resolved
Storage Drivers
AUFS
| Issue | Abstract | Impact | Reproducibility | Cause | Solution | Notes |
|---|---|---|---|---|---|---|
| :white_check_mark: #783 | Cannot access to a directory due to a permission error | :neutral_face: Medium | :smiley: Easy | Expected AUFS behavior. dirperm1 mount option fixes this issue. | Update the kernel (AUFS >= 2008xxxx?) and Docker daemon (>= 1.7) | Confirm: `docker info |
| :white_check_mark: #18180 | A process becomes a zombie and hangs up | :scream: High | :scream: Hard(multiprocessor) :smiley: Easy(uniprocessor) | Compatibility between the kernel and AUFS | Update the kernel (AUFS >= 20160111) | Java apps and MongoDB are known to be affected |
| :white_check_mark: #20199 | fcntl(F_SETFL, O_APPEND) is ignored and hence data can be corrupted | :scream: High | :smiley: Easy | AUFS bug | Update the kernel (AUFS >= 20160301) | Dovecot is known to be affected |
| :white_check_mark: #20240 | Weird permission even though dirperm1 is enabled | :neutral_face: Medium | :scream: Hard | AUFS bug | Update the kernel (AUFS >= 20160905) | |
| :white_large_square: AUFS ML 2016-03-08 | Hang up related to O_DIRECT | :scream: High | :smiley: Easy | Unanalyzed | None | Percona is known to be affected |
| :white_large_square: #24309 | Unable to remove files previously committed | :scream: High | :smiley: Easy | Unanalyzed | This article seems related, but perhaps slightly different(Japanese) | |
| :white_square_button: #34361 | AUFS + XFS hangs up | :scream: High | :smiley: Easy | AUFS bug | Update AUFS |
Non-bug issues:
- AUFS is not available in the mainline kernel．Only a few distros (Ubuntu, Boot2Docker, ..) support AUFS, but even for Ubuntu, Canonical says "AUFS will disappear".
- No support for extended attributes ("xattrs"), and might not ever get support (#1070, #8460).
rename(2)is not fully supported ( see also #aufs--overlay-common )
Overlay
| Issue | Abstract | Impact | Reproducibility | Cause | Solution | Notes |
|---|---|---|---|---|---|---|
| :white_check_mark: #10180 | RPMDB corruption | :scream: High | :neutral_face: Medium | Expected overlay behavior | Use yum-{utils,plugins-ovl}-1.1.31-33.el7 (included in RHEL 7.2) or later. Kernel patch is also available. | Linux 4.6 or later prints human-friendly dmesg |
| :white_check_mark: #12080 | Cannot use UNIX domain sockets | :neutral_face: Medium | :smiley: Easy | Overlay Bug | Use Linux 4.7-rc4 or later | |
| :white_check_mark: #12327 | pip fails | :scream: High | :smiley: Easy | Overlay Bug | Use Linux 4.5 or later | |
| :white_check_mark: #19082 | Weird behavior after removing the current directory | :smiley: Low | :smiley: Easy | Overlay Bug | Use Linux 4.5 or later | |
| :white_square_button: #19647, coreos/bugs#1095 | Untar fails intermittently | :scream: High | :scream: Hard | Overlay Bug | Use Linux 4.13 with OVERLAY_FS_INDEX=y | Analysis is in progress in coreos/bugs#1095 |
| :white_large_square: #20640 | Container cannot be started | :neutral_face: Medium | :scream: Hard | Unanalyzed | None | Possibly identical to #16902 |
| :white_check_mark: #20950 | /dev/console: operation not permitted | :scream: High | :smiley: Easy | Kernel Bug | Use recent Linux kernels | |
| :white_check_mark: #21555 | docker build fails intermittently (overlay1) | :scream: High | :scream: Hard | DiffDriver bug | Use Docker 1.13 or later | Overlay2 doesn't have this issue by design |
| :white_check_mark: #24913 | permissions broken after chown | :neutral_face: Medium | :smiley: Easy | Overlay Bug | Use Linux 4.6 or later | The overlay2 issue #28391 is due to the identical bug |
| :white_check_mark: #25244 | opaque flag not reset after directory copy up | :neutral_face: Medium | :smiley: Easy | Overlay Bug | Resolved in Linux 4.8 and backported to 4.4.21 and 4.7.4 | npm is known to be affected |
| :white_check_mark: machine#3327 | chmod fails with EPERM | :smiley: Low | :smiley: Easy | Overlay Bug | Use Linux 4.5 or later | |
| :white_check_mark:#27358 | file removal weird on overlay + XFS (ftype=0) | :scream: High | :smiley: Easy | Expected behavior | Format xfs with ftype=1 | |
| :white_check_mark:#34320 | docker build produces weird images with CONFIG_OVERLAY_FS_REDIRECT_DIR=y | :scream: High | :smiley: Easy | DiffDriver issue | Apply #34342 (Docker 17.08?) |
Non-bug issues:
- :scream: High inode usage (resolved in overlay2, which will be available in Docker 1.12)
- Red Hat says "OverlayFS remains a Technology Preview in Red Hat Enterprise Linux 7.3 under most circumstances"
rename(2)is not fully supported ( see also #aufs--overlay-common )- MySQL doesn't work without
touch-ing files under/var/lib/mysql: https://github.com/docker/for-linux/issues/72#issuecomment-319904698
AUFS / Overlay common
Non-bug issue: rename(2) is not fully supported #25409
reports about the incompatible behavior of rename(2) from the real world
BtrFS
| Issue | Abstract | Impact | Reproducibility | Cause | Solution | Notes |
|---|---|---|---|---|---|---|
| :white_check_mark: #19073 | sendfile(2) can be unkillable | :smiley: Low | :smiley: Easy | BtrFS bug | None | Not likely to happen in production, but needs consideration for public PaaS |
| :white_large_square: #20080 | cgroups kmem limit leads crash and data corruption | :scream: High | :smiley: Easy? | Btrfs bug | Avoid kmem limit configuration? |
Non-bug issues:
- Slow #10161
- No page sharing (e.g. same DLLs are loaded redundantly) http://comments.gmane.org/gmane.comp.sysutils.docker.devel/1384
- Docker says BtrFS is Experimental. Red Hat says BtrFS is Tech Preview.
ZFS
| Issue | Abstract | Impact | Reproducibility | Cause | Solution | Notes |
|---|---|---|---|---|---|---|
| :white_check_mark: #20153 | Some operations fail due to EBUSY | :neutral_face: Medium | :neutral_face: Medium | Daemon bug | Update Docker daemon |
Non-bug issues:
- Docker says ZFS is not recommended for production.
DeviceMapper
| Issue | Abstract | Impact | Reproducibility | Cause | Solution | Notes |
|---|---|---|---|---|---|---|
| :white_check_mark: #4036 | Mount fails | :scream: High | :smiley: Easy | udev sync disabled | Use a Docker daemon binary which supports udev sync | Confirm: `docker info |
| :white_large_square: #20401 | Infinite “mount/remount” loop, which makes the system unresponsive | :scream: High | :scream: High | Unanalyzed (perhaps related to XFS) | None |
Non-bug issues:
- Slow #10161
- No page sharing (e.g. same DLLs are loaded redundantly) http://comments.gmane.org/gmane.comp.sysutils.docker.devel/1384
Storage driver test tool
- dmcgowan/dsdbench: Docker Storage Driver Benchmarks and Tests
So which storage driver should I use?
It totally depends on your workload, but Docker, Inc. says AUFS and Devicemapper (direct-lvm) are "production-ready".
Although not listed in the above table, VFS driver is also attractive for its robustness.
Links:
- https://jpetazzo.github.io/assets/2015-03-03-not-so-deep-dive-into-docker-storage-drivers.html#1
- http://www.projectatomic.io/docs/filesystems/
- https://blog.jessfraz.com/post/the-brutally-honest-guide-to-docker-graphdrivers/
Anyway...
You know, containers should be "immutable" and "disposable".
For persistent data and some special temporary data, you should better consider using an external volume (docker run -v).
Links:
Network
| Issue | Abstract | Impact | Reproducibility | Cause | Solution | Notes |
|---|---|---|---|---|---|---|
| :white_square_button: #5618 | hang up with unregister_netdevice: waiting for lo to become free | :scream: High | :scream: Hard | Kernel bug | Use Linux 4.8 or later | The patch will be backported to old kernels in major distros |
| :white_check_mark: #18776 | TCP checksums are ignored | :scream: High | :scream: Hard | Kernel bug | Use Linux 4.4 or later | blog |
Logging
| Issue | Abstract | Impact | Reproducibility | Cause | Solution | Notes |
|---|---|---|---|---|---|---|
| :white_check_mark: #19209 | GELF driver saturates CPU | :scream: High | :smiley: Easy | Compression | Disable compression | |
| :white_check_mark: #18057,#20600 | cat /dev/zero leads to out of memory | :scream: High | :smiley: Easy | logger's stdio handling issue | Use Docker 1.13 or later (or just disable the logging) | Related: #21181 |
| :white_large_square: #22497 | container cannot be stopped if many logs are being printed | :scream: High | :scream: Hard | logger's stdio handling issue | ||
| :white_check_mark: #22502 | logging blocks the container | :scream: High | :smiley: Easy | logger's stdio handling issue | Use Docker 1.11 or later | affected versions: 1.10.0 |
Others
| Issue | Abstract | Impact | Reproducibility | Cause | Solution | Notes |
|---|---|---|---|---|---|---|
| :white_check_mark: #17720 | Docker daemon 1.9 serious performance issue | :scream: High | :scream: Hard | ? | Use Docker 1.10 | |
| :white_large_square: #19758 | soft lockup related to show_mountinfo(), after frequent docker run | :scream: High | :scream: Hard | Unanalyzed (Kernel bug related to the number of processors?) | None | |
| :white_check_mark: #20670 | /dev/pts unmounted on the HOST when you are using -v /dev:/dev (After that you can no longer open SSH nor xterm) | :scream: High | :smiley: Easy | daemon bug related to mount namespace | Use Docker 1.11.1. (Or Spawn the docker daemon from systemd. Or do not use -v /dev:/dev) | |
| :white_check_mark: #20836 | Daemon hangs up after frequent docker run | :scream: High | :scream: Hard | Daemon bug | Use Docker 1.11.1 | |
| :white_check_mark: #28936 | Strange permission issues with named containers on 1.12.3 | :scream: High | :smiley: Easy | Daemon bug related to SELinux) | Use Docker 1.12.4 | |
| :white_check_mark: Ubuntu linux-azure #1719045 | fatal error: unaligned sysUnused on Azure | :scream: High | ? | Ubuntu linux-azure kernel bug | Use linux-azure 4.11.0-1013.13 or later |
Non-bug issues: