Join us
@squadcast ă» Aug 17,2023 ă» 13 min read ă» 1681 views ă» Originally posted on www.squadcast.com
Explore Docker container logs best practices and gain insight into debugging and troubleshooting techniques using Docker Compose.
Docker Compose is a tool for defining and running multi-container Docker applications. It allows developers to streamline the process of configuring, building, and running multiple containers as a single unit with a docker-compose.yml. This configuration file specifies the services, networks, and volumes required for an application, and their relationships and dependencies.
The docker-compose logs command displays the logs of all services defined in the docker-compose.yml file. It helps monitor and debug applications by providing insights into the behavior and performance of the various services.
The command aggregates the logs from all the containers specified in the docker-compose.yml file, presenting them in a unified view. By default, the logs are shown in the order they were generated, but you can filter or customize the output using various flags and options, such as:
This article will explore Docker Compose logging drivers and logging strategy best practices, practical examples of debugging and troubleshooting using Docker logs, and demonstrate how to set up log streaming.
The table below summarizes key Docker Compose logs concepts this article will build upon.
Concept: docker-compose logs
Description:docker-compose logs is a Docker command used to view the container logs for a systemâs defined services.
Concept: Logging drivers
Description:Docker supports several logging drivers that define how your container logs are collected and stored.
Concept:Logging strategies
Description:There are two log delivery modes in Docker: blocking and non-blocking.
Concept:Debugging with logs
Description:The docker logs command allows you to inspect specific containers and review logs that could provide insight into the issue your application is facing.
Concept:Storing logs
Description:Maintaining a healthy system requires a clear understanding of log locations and adherence to lifecycle policy guidelines.
Logging drivers are plugins that handle container logs in Docker. They define how logs are collected, processed, and stored for a container. Each driver provides different features and is designed to work with various logging services and platforms.
The default logging driver in Docker Compose is the json-file driver. This driver stores logs as JSON files on the host machine where the container is running. However, Docker supports several other logging drivers you can configure in your Docker Compose setup.
To configure Docker Compose to use separate logging drivers, specify the desired driver in the docker-compose.yml file using the logging configuration option for each service. Here's an example:
version: '3'
services:
web:
image: my-web-app
logging:
driver: gelf
options:
gelf-address: "udp://logstash-host:12201"
tag: "my-web-app"
db:
image: my-db
logging:
driver: fluentd
options:
fluentd-address: "fluentd-host:24224"
tag: "my-db"
In this example, the web service is configured to send logs to a Logstash server in the ELK stack using the gelf logging driver. The gelf-address option is set to the address of the Logstash server, which is configured to listen for GELF input on port 12201 (you need to configure Logstash accordingly).
The db service is configured to send logs to a Fluentd server using the fluentd logging driver. The fluentd-address option is set to the address of the Fluentd server, which listens on port 24224 (default Fluentd port).
Before using this configuration, ensure you have the ELK stack and Fluentd servers set up and properly configured to receive logs from your Docker containers.
The log delivery mode in Docker determines how logs are transferred from the running containers to the specified log driver. There are two delivery modes:
To understand the risk of log loss with non-blocking mode, suppose you have a Docker Compose file (docker-compose.yml) that defines two services: a web server and a database. The web server logs important information to stdout or stderr within the container, and you expect these logs to be available for debugging or monitoring purposes.
version: '3'
services:
web:
build: .
ports:
- 8080:80
db:
image: mysql:latest
environment:
- MYSQL_ROOT_PASSWORD=password
If you start the services using docker-compose up -d in non-blocking mode (detached mode), the containers will run in the background. However, if you don't actively capture or redirect the logs to a file or logging system, the logs will not be immediately visible or persisted. This can lead to potential log loss.
There is no one-size-fits-all Docker Compose logging strategy. However, several well-established best practices can help you define the right strategy for specific use cases. Here are four key Docker Compose logging best practices to consider:
Integrated full stack reliability management platformTry For Free Drive better business outcomes with incident analytics, reliability insights, SLO tracking, and error budgets Manage incidents on the go with native iOS and Android mobile apps Seamlessly integrated alert routing, on-call, and incident response
Debugging a containerized application using Docker Compose logs can be very efficient, especially when dealing with HTTP 500 error codes. HTTP 500 is a generic error message indicating that the server encountered an unexpected condition that prevented it from fulfilling the request.
Here is a step-by-step guide for debugging HTTP 500 errors with Docker Compose logs.
docker-compose ps
docker logs <container-id>
docker logs <container-id> 2>&1 | grep "500"
This will display only the logs that contain the keyword "500".
docker logs -f <container-id>
docker logs âfollow <container-id>
This will continuously display the logs as they are generated. Look for any patterns or recurring error messages.
docker inspect <container-id>
Logs in Docker are typically stored on the host system where the Docker daemon runs. The exact location and format depend on the Docker logging driver. For example, if you're using the default json-file driver, the logs are stored in JSON format at the following location:
/var/lib/docker/containers/<container-id>/<container-id>-json.log
If you're using a different logging driver, like syslog or journald, the logs are stored in the location determined by that system's configuration.
Managing logs is a critical task in maintaining the health of your system. Here are some guidelines for creating a lifecycle policy based on the aggregate size of logs:
If you're working with a multi-container Docker environment and experiencing issues, Docker Compose logs can be extremely useful for troubleshooting. Below is a step-by-step guide on troubleshooting with Docker Compose logs using a real-world example of a Python Flask application with a PostgreSQL database.
Assume your docker-compose.yml file looks like this:
version: '3'
services:
web:
build: .
command: python app.py
volumes:
- .:/code
ports:
- '5000:5000'
depends_on:
- db
db:
image: postgres:11
environment:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
POSTGRES_DB: test_db
Suppose you notice that your application is not responding as expected, and you suspect an issue with the database connection. You can follow the steps below to debug.
First, check the logs for the web service, which is where the application is running:
If you see an error message related to the database connection, such as:
OperationalError: (psycopg2.OperationalError) could not connect to server: Connection refused
This indicates that the application is unable to connect to the database.
Next, check the logs for the database service to see if there are any issues there:
docker-compose logs db
Based on the error message, you may have specified the wrong password for the PostgreSQL user in your docker-compose.yml file. In this case, you'd need to correct the password and run docker-compose up again.
If you're still experiencing issues, following the logs in real-time as you interact with the application could be helpful. You can do this with the -f or --follow option:
docker-compose logs -f
You can filter the logs using grep to find a specific error or message. For example, to find log entries that contain the word 'error', you could run:
docker-compose logs web | grep error
In this example, we provide a sample application demonstrating log streaming between containers and why it is important. We have two components:
The client sends a POST request to the server to calculate the Fibonacci sequence. The server then computes the sequence and sends back the response. This simple model could be further extended to distribute all sorts of workloads as microservices.
The following docker-compose.yml file represents the setup described above.
version: '3'
services:
redis:
image: redis:5
ports:
- "6379:6379"
web:
build: ./server
command: flask run --host=0.0.0.0 --port=5001
volumes:
- ./server:/code
ports:
- "5001:5001"
environment:
- FLASK_APP=app_server.py
- FLASK_RUN_HOST=0.0.0.0
- CELERY_BROKER_URL=redis://redis:6379/0
- CELERY_RESULT_BACKEND=redis://redis:6379/1
depends_on:
- redis
worker:
build: ./server
# command: celery -A tasks.celery worker --loglevel=info
command: celery -A tasks worker --loglevel=info
volumes:
- ./server:/code
environment:
- CELERY_BROKER_URL=redis://redis:6379/0
- CELERY_RESULT_BACKEND=redis://redis:6379/1
depends_on:
- web
client:
build:
context: ./client
dockerfile: Dockerfile
volumes:
- ./client:/code
depends_on:
- worker
The docker-compose.yml file above creates four containers: redis, worker, and web constitute the server component, and the client is a separate component. For simplicity's sake, to avoid â404 not foundâ errors, we added the `depends_on` key to the client image so that the client sends the request only after the server is fully up.
Once all the containers are up, the client sends a POST request with a single parameter `n` to compute Fibonacci sequence up until n Fibonacci numbers. For example, if the parameter `n` is 10, the sequence would be [0, 1, 1, 2, 3, 5, 8, 13, 21, 34]. Each Fibonacci number the server calculates is sent to the client as a log emission.
This behavior is helpful when a client issues a long-running command to the server, and instead of waiting at a blank screen, the server can emit feedback logs.
Celery workers are used to hand off compute-intensive or long-running tasks that can be processed asynchronously. Celery requires a message broker - in this case, Redis - to facilitate communication between the task producer (the client application) and the task consumer (the worker)
The complete code for the application above is available at:
https://github.com/ashaik4/distributed_task_framework
Docker Compose enables the configuration, building, and running of multi-container Docker applications using a YAML configuration file, docker-compose.yml. Developers can monitor applications by displaying logs of all defined services with the docker-compose logs command.
Docker supports various logging drivers that process and store container logs. Docker's log delivery modes, 'blocking' and 'non-blocking,' affect how logs are transferred from containers to the specified log driver. Debugging with logs can be efficient in identifying issues, such as âHTTP 500â error codes.
Logs are stored based on the logging driver used, and it's essential to have a log lifecycle policy to manage the aggregate size of logs. Docker Compose logs are also useful for troubleshooting in a multi-container environment. Real-time log streaming can provide client feedback for long-running tasks processed asynchronously by server-side Celery workers.
Join other developers and claim your FAUN account now!
Influence
Total Hits
Posts
Only registered users can post comments. Please, login or signup.