Docker container logs taking up a lot of disk space

Site Operators
1

We noticed that Docker container logs were taking up a significant about of disk space. I couldn’t find anywhere in tutor configuration of setting a --log-opt max-size=50m for example.

Has anyone run into this issue?

image
image779×500 68.9 KB

We ran this command to see the total disk usage for these Docker containers. It looks like 66 GB used.

(venv) ubuntu@ip-172-16-17-211:~$ sudo sh -c "du -ch /var/lib/docker/containers/*/*-json.log"
4.0M    /var/lib/docker/containers/2aed8a0b7562ec52486c073c378035de0774806542ebf6bca514420863ac9a5a/2aed8a0b7562ec52486c073c378035de0774806542ebf6bca514420863ac9a5a-json.log
16K     /var/lib/docker/containers/527aeb5ed435596de5ebeddb724c85f2f8d01621525da6782b4e36c239c6b670/527aeb5ed435596de5ebeddb724c85f2f8d01621525da6782b4e36c239c6b670-json.log
88K     /var/lib/docker/containers/642e24676db53988cadc3a79a8a1dd63684bed347a0a4d3fcfe2c2883a2aca75/642e24676db53988cadc3a79a8a1dd63684bed347a0a4d3fcfe2c2883a2aca75-json.log
12G     /var/lib/docker/containers/73fefb5d24387cf4fb7045caaad5bc11cbf0a5b0c784595e59e09f04be0a3870/73fefb5d24387cf4fb7045caaad5bc11cbf0a5b0c784595e59e09f04be0a3870-json.log
1.1G    /var/lib/docker/containers/818957e549e3eea97b9b8f08e8dc218f318c7f4f7069e53b8c2235216d2e869e/818957e549e3eea97b9b8f08e8dc218f318c7f4f7069e53b8c2235216d2e869e-json.log
49G     /var/lib/docker/containers/978a2f99e87dde426bea5f85bdd2e017760403029a1f0336f7fd314ddd5d4398/978a2f99e87dde426bea5f85bdd2e017760403029a1f0336f7fd314ddd5d4398-json.log
16K     /var/lib/docker/containers/a47a04b91cb9a9fdcd6ffa5e6fec47fb85ed9ccdfc5065cdafdaac45a6b0e218/a47a04b91cb9a9fdcd6ffa5e6fec47fb85ed9ccdfc5065cdafdaac45a6b0e218-json.log
88K     /var/lib/docker/containers/a86dff6ec11fdb756b1009fd07a743a1e2e062cf01152fc2ec255c10eb060b29/a86dff6ec11fdb756b1009fd07a743a1e2e062cf01152fc2ec255c10eb060b29-json.log
144M    /var/lib/docker/containers/acf8e7af265b15f29be6de5bd0679fd7d57eb8f27ecb3e059e0d04d3a7621c9f/acf8e7af265b15f29be6de5bd0679fd7d57eb8f27ecb3e059e0d04d3a7621c9f-json.log
41M     /var/lib/docker/containers/afa9f78485fef6875505afb7daf8f2aa68ec24d62cfbd1c260588bde1521d161/afa9f78485fef6875505afb7daf8f2aa68ec24d62cfbd1c260588bde1521d161-json.log
3.5G    /var/lib/docker/containers/c6221011ae032af3d6c71c2bfc7660e3aa6ffff0ffec5b6cf149320a83fc266f/c6221011ae032af3d6c71c2bfc7660e3aa6ffff0ffec5b6cf149320a83fc266f-json.log
76K     /var/lib/docker/containers/e28349bc76007cb0fa3b37b01d8be7f96fe9335433376a5700bb66ae8d762090/e28349bc76007cb0fa3b37b01d8be7f96fe9335433376a5700bb66ae8d762090-json.log
92K     /var/lib/docker/containers/effafc40fde03891106fc0886d38e488f94548ed578fd8ce060c49f75a60f82c/effafc40fde03891106fc0886d38e488f94548ed578fd8ce060c49f75a60f82c-json.log
248M    /var/lib/docker/containers/fe1f9ee5923bda725b4b05c7d0b382899fcc056c839579822f7e6c7d49ade7ad/fe1f9ee5923bda725b4b05c7d0b382899fcc056c839579822f7e6c7d49ade7ad-json.log
66G     total

cc @regis @dave @braden

2

When trying to configure the Docker daemon settings I couldn’t find the Daemon | Docker Docs location at /etc/docker/daemon.json or ~/.config/docker/daemon.json.

Looking at this article it recommends creating file and updating the /lib/systemd/system/docker.service file to include the --config-file=/etc/docker/daemon.json path.

I realize that not everyone is running tutor local but we’re still doing this currently. Here are our specs for this EC2 instance.

Operating System

(venv) ubuntu@ip-172-16-17-211:~$ cat /etc/os-release 
PRETTY_NAME="Ubuntu 22.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.1 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

Docker Info

(venv) ubuntu@ip-172-16-17-211:~$ docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Docker Buildx (Docker Inc., v0.9.1-docker)
  compose: Docker Compose (Docker Inc., v2.12.2)
  scan: Docker Scan (Docker Inc., v0.21.0)

Server:
 Containers: 18
  Running: 12
  Paused: 0
  Stopped: 6
 Images: 129
 Server Version: 20.10.21
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 770bd0108c32f3fb5c73ae1264f7e503fe7b2661
 runc version: v1.1.4-0-g5fd4c4d
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 6.8.0-1015-aws
 Operating System: Ubuntu 22.04.1 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 61.78GiB
 Name: ip-172-16-17-211
 ID: C2A2:33J5:TLF6:YAHW:N6RO:PGTD:CXKU:DV6B:F63Z:FXVO:HP74:3G47
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

I see a lot of processes running the Docker daemon with no --config-file=/etc/docker/daemon.json at the moment.

image
image1188×644 196 KB

Solution
Let me know if you think this would be a good solution to limiting the docker container logs from increasing past 50 MB in size.

  1. Create file at /etc/docker/daemon.json and add the following.
{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "500m",
    "max-file": "2",
    "compress": "false"
  }
}
  1. Update /lib/systemd/system/docker.service by adding in the --config-file=/etc/docker/daemon.json to the dockerd command for ExecStart.
  2. Restart docker daemon with systemctl daemon-reload.
3

This issue is documented here: Docker container logs fill up disk space on production servers · Issue #1014 · overhangio/tutor · GitHub

4

Thanks @regis! I’ll look over that issue. Everything about configuring the Docker daemon JSON File logging driver can be found here.

There is a note about this so that is important.

Restart Docker for the changes to take effect for newly created containers. Existing containers don’t use the new logging configuration automatically.

5

@regis
You mention in this comment Docker container logs fill up disk space on production servers · Issue #1014 · overhangio/tutor · GitHub that you would recommend configuring the Docker daemon logging configuration for setting up log rotations. I think this may be the straightforward approach to handling container log configuration that even falls outside of the typical tutor local deployment. Setting the Docker daemon logging configuration is most likely what we’ll end up doing for our site.

Docker Container Logging Configuration
Here are some things that I noticed though after reading over Docker documentation here Services top-level elements | Docker Docs. We do have the ability to set the docker-compose service logging element to define overrides. I believe that the container logging configuration would take precedence over the Docker daemon for logging.

image
image662×359 26.4 KB

Would it be better to utilize docker-compose service configuration to include this logging configuration rather than setting Docker daemon logging configuration? I honestly don’t mind setting the Docker daemon logging configuration file because it is a single change versus having to handle this on a container-level.

Update: After reading this article I think that Docker container logging configuration would be useful if you were utilizing different logging configuration per service rather than the Docker logging default drivers for all containers. This article explains all that Docker Compose Logs: Best Practices & Expert Troubleshooting Guide | Squadcast. Because we are not at the moment going to set up different logging drivers, setting the Docker daemon would make more sense to configure.

6

@regis
After making the following Docker daemon configuration changes on my staging environment it appears that the containers are set up properly for docker logging configuration.

Docker Daemon Logging Configuration Changes

# Do this from the `root` account.

# /etc/docker/daemon.json 
{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m",
    "max-file": "10",
    "compress": "false"
  }
}

# /lib/systemd/system/docker.service
# Edit the `ExecStart` command by adding in the `--config-file` option.
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --config-file=/etc/docker/daemon.json

# Restart Docker Daemon configuration changes.
# Ref: https://labex.io/tutorials/linux-linux-systemctl-daemon-reload-390500
sudo systemctl restart docker

Stop and Rebuild Docker Containers for tutor local
The containers need to be recreated to be able to get the new log configuration applied by the Docker daemon configuration.

tutor local stop
docker container prune
tutor local start -d --skip-build

Verifying Docker Container Configuration

# Checking container log configuration.
docker inspect -f '{{.HostConfig.LogConfig}}' tutor_local_cms_1
docker inspect -f '{{.HostConfig.LogConfig}}' tutor_local_cms-worker_1
docker inspect -f '{{.HostConfig.LogConfig}}' tutor_local_lms_1
docker inspect -f '{{.HostConfig.LogConfig}}' tutor_local_lms-worker_1
docker inspect -f '{{.HostConfig.LogConfig}}' tutor_local_caddy_1
docker inspect -f '{{.HostConfig.LogConfig}}' tutor_local_discovery_1
docker inspect -f '{{.HostConfig.LogConfig}}' tutor_local-mfe-1

# output from the docker inspect commands.
{json-file map[compress:false max-file:10 max-size:100m]}
{json-file map[compress:false max-file:10 max-size:100m]}
{json-file map[compress:false max-file:10 max-size:100m]}
{json-file map[compress:false max-file:10 max-size:100m]}
{json-file map[compress:false max-file:10 max-size:100m]}
{json-file map[compress:false max-file:10 max-size:100m]}
{json-file map[compress:false max-file:10 max-size:100m]}

Checking that the dockerd processes are using the --config-file.

image
image1510×301 101 KB