The logs visible with docker logs do not continuously accumulate in system memory. Instead, they are saved as files on the disk of the host system by default.

Where and how are the logs stored? ⚙️

Docker captures both the standard output (stdout) and standard error (stderr) streams of the container and processes them through a logging driver. Unless specified otherwise, Docker uses the json-file logging driver by default.

  • Storage location: The logs for the container are saved as JSON-formatted files at a specific path in the host system. Typically, the path is as follows:

    • /var/lib/docker/containers/[container_ID]/[container_ID]-json.log

  • The docker logs command: This command does not actually show content residing in memory; instead, it reads the JSON log file mentioned above and displays it in the terminal.

Doesn’t the continuous accumulation of logs lead to insufficient disk space? 🤔

Yes, that's correct. If no actions are taken, the log files can grow indefinitely to occupy all available disk space on the host. To prevent this, Docker offers a log rotation option to limit the size and number of log files.

For example, when executing the docker run command, you can add the following options:

docker run -d \
  --log-driver json-file \
  --log-opt max-size=10m \
  --log-opt max-file=3 \
  nginx

  • --log-opt max-size=10m: Limits the maximum size of a single log file to 10MB.

  • --log-opt max-file=3: Limits the number of log files to a maximum of 3. If it exceeds 3, the oldest file will be deleted first.

With this setting, the logs of the specified container will use a maximum of 30MB (10MB * 3) of disk space.

Conclusion 📝

  • The console logs of Docker containers are saved as files on the host disk rather than system memory.

  • The docker logs command serves to read and display these stored files.

  • It is very important to set the log rotation options (max-size and max-file) to prevent log files from growing indefinitely.

In Docker Compose, you can easily configure logging options for each service within the docker-compose.yml file.

You can think of the --log-opt flag in the docker run command as being in YAML format.

Setting in docker-compose.yml file 📝

Below is an example of applying log rotation to a service named my-app.

services:
  my-app:
    image: your-app-image
    # ... other settings ...
    deploy:
      resources:
        limits:
          memory: 512M

    # 👇 This is the log configuration.
    logging:
      driver: "json-file"  # Logging driver to be used (default, can be omitted)
      options:
        max-size: "10m"    # Maximum size per file: 10MB
        max-file: "3"      # Maximum number of files: 3

  • logging: This is the key that starts the logging configuration for that service.

  • driver: Specifies the logging driver to be used. json-file is the default.

  • options: Sets options to be passed to the driver.

    • max-size: Equivalent to --log-opt max-size in docker run.

    • max-file: Equivalent to --log-opt max-file in docker run.

Now, when you start this service with docker-compose up, the logs for the my-app container will automatically rotate, using only a maximum of 30MB (10MB * 3) of disk space.

Setting default for all containers 🌍

If it’s cumbersome to specify options for each container and you want to apply a default policy across the entire system, modifying the Docker daemon configuration file is the best approach.

  1. Open or create the /etc/docker/daemon.json file. (Create a new file if it doesn’t exist.)

  2. Write the contents as follows:

{
    "log-driver": "json-file",
    "log-opts": {
        "max-size": "10m",
        "max-file": "3"
    }
}
  1. Restart the Docker daemon.
sudo systemctl restart docker

This configuration ensures that all new containers created will automatically adhere to this policy unless specific logging configurations are defined. Of course, if particular logging options are specified in the docker-compose.yml file or the docker run command for individual containers, those settings take precedence over the daemon configuration.

Summary

  • Individual service settings: Configuring within the docker-compose.yml file's logging section is the most common and convenient method.

  • Global default settings: To apply a uniform policy to all containers on the server, modify the /etc/docker/daemon.json file.