Springboot + ELK + Docker

Example of implementing an API using Spring Boot, Elasticsearch, Logstash, Kibana, and Docker.

In this article, we will cover an example of an application that uses Springboot + ELK + Docker. It is a simple application, only intended to demonstrate the concepts. I won’t go into a detailed explanation of all the elements (if you’re not familiar with this stack, I suggest doing a little research before doing this tutorial).

Pre-requirements

You must have docker installed on your machine.
An IDE of your choice (Intellij, Eclipse, or Vscode)
Some knowledge of Springboot, Elasticsearch, Logstash, Kibana, and Docker

Hands-On

Springboot Application

I’ll start with the application; I created a REST API with 4 endpoints: This is how the application’s Dockerfile was:

                FROM openjdk:17.0.1 
WORKDIR /app 
COPY .mvn/ .mvn 
COPY mvnw pom.xml ./ 
RUN ./mvnw dependency:go-offline 
COPY src ./src 
CMD ["./mvnw", "spring-boot:run"] 
EXPOSE 8080

Alternatively, you can use a Dockerfile like this:

                FROM openjdk:17.0.1 
WORKDIR /app 
COPY ./target/*.jar ./app.jar 
ENTRYPOINT ["java", "-jar", "/app/app.jar"]
 
EXPOSE 8080

As we are going to use several containers, we are going to use docker compose. At the root of your project, you will create the docker-compose.yml file. For now, docker-compose.yml will look like this.

                version: '3.2' 
  services: 
  app: 
    container_name: library_app 
    build: 
       context: . 
    ports: - "8080:8080"

For now, docker-compose.yml will look like this.

To test, run from the root of the project: docker-compose up

Test by calling the endpoints via Postman, browser, or others of your choice.

Elasticsearch

I won’t go into details about Elasticsearch (I suggest you look into it if you don’t know about it).

                elasticsearch: 
  container_name: library_elasticsearch 
  image: docker.elastic.co/elasticsearch/elasticsearch:7.15.2 
  volumes: 
    - type: volume 
    source: elasticsearch 
    target: /usr/share/elasticsearch/data 
  ports: - "9200:9200" - "9300:9300" 
  environment: 
    ES_JAVA_OPTS: "-Xmx256m -Xms256m" 
    ELASTIC_PASSWORD: changeme 
    discovery.type: single-node 
networks: - elk 
networks: 
  elk: 
    driver: bridge 
volumes: 
  elasticsearch

Here we are setting Elasticsearch settings like port, memory variables, base directory in docker, etc. We also created a network called elk and added our service to it. To test, run: docker-compose up, open this URL http://localhost:9200/, and you will have a similar result to this:

This shows us that Elasticsearch is working correctly.

Logstash

For Logstash, the process is a little different: first let’s create a folder called .logstash, to store some settings. Inside it, we will create the logstash.conf file, which will have the following information:

                input { 
    tcp { 
      mode => "server" 
      port => 4560 
      codec => json_lines 
    } 
    file { 
      type => "java" 
      path => "/var/log/logs/library/application.log" 
      codec => multiline { 
      pattern => "^%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{TIME}.*" 
      negate => "true" 
      what => "previous" 
    } 
  } 
} 
output { 
    stdout { 
       codec => rubydebug 
    } 
    elasticsearch { 
       index => "library-logstash-%{+YYYY.MM.dd}" 
       hosts => "elasticsearch:9200" 
       user => "elastic" 
       password => "changeme" 
       ecs_compatibility => disabled 
   } 
}

Here we will have the operating mode, which can be TCP or per file. In TCP mode, logstash will get real-time data from the port specified in the logback-spring.xml file, inside the project in the resources package.

logback-spring.xml file:

                <?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <include resource="org/springframework/boot/logging/logback/defaults.xml"/>
    <include resource="org/springframework/boot/logging/logback/base.xml"/>
    <springProperty scope="context" name="appName" source="spring.application.name"/>
    <property name="LOG_FILE" value="${BUILD_FOLDER:-build}/${appName}"/>
    <property name="CONSOLE_LOG_PATTERN"
              value="%clr(%d{yyyy-MM-dd HH:mm:ss.SSS}){faint} %clr(${LOG_LEVEL_PATTERN:-%5p}) %clr(${PID:- }){magenta} %clr(---){faint} %clr([%15.15t]){faint} %m%n${LOG_EXCEPTION_CONVERSION_WORD:-%wEx}}"/><appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
        <filter class="ch.qos.logback.classic.filter.ThresholdFilter">
            <level>INFO</level>
        </filter>
        <encoder>
            <pattern>${CONSOLE_LOG_PATTERN}</pattern>
            <charset>utf8</charset>
        </encoder>
    </appender><appender name="LOGSTASH" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
        <destination>logstash:4560</destination>
        <encoder charset="UTF-8" class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
            <providers>
                <timestamp>
                    <timeZone>UTC</timeZone>
                </timestamp>
                <pattern>
                    <pattern>
                        {
                        "logLevel": "%level",
                        "serviceName": "${springAppName:-}",
                        "pid": "${PID:-}",
                        "thread": "%thread",
                        "class": "%logger{40}",
                        "rest": "%message"
                        }
                    </pattern>
                </pattern>
            </providers>
        </encoder><encoder class="net.logstash.logback.encoder.LogstashEncoder">
            <includeCallerData>true</includeCallerData>
        </encoder>
    </appender><appender name="STASH" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>logback/redditApp.log</file>
        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <fileNamePattern>logback/redditApp.%d{yyyy-MM-dd}.log</fileNamePattern>
            <maxHistory>7</maxHistory>
        </rollingPolicy>
        <encoder class="net.logstash.logback.encoder.LogstashEncoder"/>
    </appender><root level="INFO">
        <appender-ref ref="LOGSTASH"/>
        <appender-ref ref="CONSOLE"/>
        <appender-ref ref="FILE"/>
        <appender-ref ref="STASH"/>
    </root>
</configuration><include resource="org/springframework/boot/logging/logback/base.xml"/><springProperty scope="context" name="appName" source="spring.application.name"/>

It also contains the output which, in our case, will be for Elasticsearch, as described below. In the output information, we define the destination, elastic user and password (in this case it is with the default values), and index. The index will serve to filter the information only from this application in kibana. In docker-compose.yml, the container will look like this:

                logstash: 
  container_name: library_logstash 
  image: docker.elastic.co/logstash/logstash:7.15.2 
  volumes: 
    - type: bind 
      source: .logstash 
      target: /usr/share/logstash/pipeline 
      read_only: true 
  ports: 
    - "5044:5044" 
    - "5000:5000/tcp" 
    - "5000:5000/udp" 
    - "9600:9600" 
    - "4560:4560" 
  environment: 
    LS_JAVA_OPTS: "-Xmx256m -Xms256m" 
  networks: 
     - elk 
  depends_on: 
     - elasticsearch

Kibana

Kibana has its simple configuration, just add to docker-compose.yml:

                kibana: 
  container_name: library_kibana 
  image: docker.elastic.co/kibana/kibana:7.15.2 
  ports: 
    - "5601:5601" 
  networks: 
    - elk 
  depends_on: 
    - elasticsearch

After that, it is now possible to access kibana through the browser at URL: http://localhost:5601/. When using kibana, you will need to add the index we created earlier in logstash to get the information. To do this, access the kibana and enter: http://localhost:5601/app/management/kibana/indexPatterns Right menu/ Stack Management/ Index Patterns/ Create index pattern. As an example, the application we cited as an example was: library-logstash-*.

Code available at: https://github.com/jefsterjr/library/tree/main

Originally published at http://github.com.