Setting Up the ELK Stack on AWS EC2 for Log Monitoring

A

Atul Vishwakarma

Guest
Welcome to this comprehensive guide on setting up the ELK Stack (Elasticsearch, Logstash, Kibana) on AWS EC2 instances. If you're managing applications in the cloud, especially Java-based ones, efficient log monitoring is crucial for debugging, performance analysis, and security. The ELK Stack, combined with Filebeat, provides a powerful, open-source solution for collecting, processing, visualizing, and analyzing logs in real-time.

In this blog post, we'll walk through a step-by-step setup using Ubuntu-based EC2 instances. This tutorial is based on a practical example involving a Java application, but the principles apply broadly. We'll cover everything from infrastructure provisioning to creating dashboards in Kibana. By the end, you'll have a fully functional log monitoring system.

Prerequisites:

  • An AWS account with access to EC2.
  • Basic knowledge of SSH, Linux commands, and AWS networking (e.g., security groups).
  • Three EC2 instances (t3.micro or similar): One for the ELK server (Elasticsearch, Logstash, Kibana), one for the client machine (hosting the app and Filebeat), and an optional web server for testing.
  • Ensure ports like 9200 (Elasticsearch), 5044 (Logstash), and 5601 (Kibana) are open in your security groups.

1. Overview of the ELK Stack​


The ELK Stack is a collection of tools from Elastic:

  • Elasticsearch: A search and analytics engine that stores and indexes your logs.
  • Logstash: A data processing pipeline that ingests, transforms, and sends logs to Elasticsearch.
  • Kibana: A web interface for visualizing and querying your logs.
  • Filebeat: A lightweight shipper that forwards logs from your application servers to Logstash.

This setup allows you to centralize logs from distributed systems like EC2, parse them into structured data, and gain insights through dashboards.

2. Infrastructure Setup​


Launch three Ubuntu EC2 instances:

  1. ELK Server: Hosts the core ELK components. Assign a public IP for Kibana access.
  2. Client Machine: Runs your Java app and Filebeat. Use the ELK server's private IP for communication.
  3. Web Server (Optional): For simulating additional log sources.

Connect via SSH to each instance as the ubuntu user. Update packages on all machines:


Code: 
sudo apt update

3. Step-by-Step Installation​

Step 1: Install and Configure Elasticsearch (on ELK Server)​


Elasticsearch is the backbone for storing logs.


  1. Install Java (required for Elasticsearch and Logstash):

    Code: 
    sudo apt update && sudo apt install openjdk-17-jre-headless -y

  2. Install Elasticsearch:

    Code: 
    wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-7.x.list

sudo apt update

sudo apt install elasticsearch -y

  3. Configure Elasticsearch: Edit /etc/elasticsearch/elasticsearch.yml:

    Code: 
    sudo vi /etc/elasticsearch/elasticsearch.yml

    Add or modify:

    Code: 
    network.host: 0.0.0.0
[cluster.name](http://cluster.name/): my-cluster
[node.name](http://node.name/): node-1
discovery.type: single-node

  4. Start and enable the service:

    Code: 
    sudo systemctl start elasticsearch
sudo systemctl enable elasticsearch
sudo systemctl status elasticsearch

  5. Verify it's running:

    Code: 
    curl -X GET "[http://localhost:9200](http://localhost:9200/)"


Step 2: Install and Configure Logstash (on ELK Server)​


Logstash processes incoming logs.


  1. Install Logstash:

    Code: 
    sudo apt install logstash -y

  2. Configure Logstash: Edit /etc/logstash/conf.d/logstash.conf:

    Code: 
    sudo vi /etc/logstash/conf.d/logstash.conf

    Add:

    Code: 
    input {
  beats {
    port => 5044
  }
}

filter {
  grok {
    match => { "message" => "%{TIMESTAMP_ISO8601:log_timestamp} %{LOGLEVEL:log_level} %{GREEDYDATA:log_message}" }
  }
}

output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "logs-%{+YYYY.MM.dd}"
  }
  stdout { codec => rubydebug }
}

    This config accepts logs from Beats (like Filebeat), parses them using Grok, and stores them in Elasticsearch.




  3. Start and enable the service:

    Code: 
    sudo systemctl start logstash
sudo systemctl enable logstash
sudo systemctl status logstash

  4. Allow traffic on port 5044:

    Code: 
    sudo ufw allow 5044/tcp

Step 3: Install and Configure Kibana (on ELK Server)​


Kibana provides the UI for log analysis.


  1. Install Kibana:

    Code: 
    sudo apt install kibana -y

  2. Configure Kibana: Edit /etc/kibana/kibana.yml:

    Code: 
    sudo vi /etc/kibana/kibana.yml

    Modify:

    Code: 
    server.host: "0.0.0.0"
elasticsearch.hosts: ["http://localhost:9200"]

  3. Start and enable the service:

    Code: 
    sudo systemctl start kibana
sudo systemctl enable kibana
sudo systemctl status kibana

  4. Allow traffic on port 5601:

    Code: 
    sudo ufw allow 5601/tcp

  5. Access the Kibana dashboard: Open a browser and navigate to http://<ELK_Server_Public_IP>:5601.


Step 4: Install and Configure Filebeat (on Client Machine)​


Filebeat ships logs from your app to Logstash.


  1. Install Filebeat:

    Code: 
    wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-7.x.list

sudo apt update 

sudo apt install filebeat -y

  2. Configure Filebeat: Edit /etc/filebeat/filebeat.yml:

    Code: 
    sudo vi /etc/filebeat/filebeat.yml

    Modify:

    Code: 
    filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - /home/ubuntu/JavaApp/target/app.log

output.logstash:
  hosts: ["<ELK_Server_Private_IP>:5044"]

  3. Start and enable the service:

    Code: 
    sudo systemctl start filebeat
sudo systemctl enable filebeat
sudo systemctl status filebeat

  4. Verify Filebeat:

    Code: 
    sudo filebeat test output


Step 5: Deploy a Java Application and Generate Logs (on Client Machine)​


To test, we'll use a sample Java app.


  1. Install Java if needed:

    Code: 
    sudo apt install openjdk-17-jre-headless -y

  2. Download and run a sample app:

    Code: 
    wget https://repo1.maven.org/maven2/org/springframework/boot/spring-boot-sample-simple/1.4.2.RELEASE/spring-boot-sample-simple-1.4.2.RELEASE.jar -O app.jar
nohup java -jar app.jar > /home/ubuntu/JavaApp/target/app.log 2>&1 &

  3. Generate test logs:

    Code: 
    echo "Test log entry $(date)" >> /home/ubuntu/JavaApp/target/app.log

Step 6: View and Analyze Logs in Kibana​

  1. In Kibana, go to Discover.
  2. Select the logs-* index pattern.
  3. Search for logs from your app, e.g., log.file.path: "/home/ubuntu/Boardgame/target/app.log".
  4. View parsed fields like log_timestamp, log_level, and log_message.



Create visualizations:

  • Pie Chart: For log level distribution.
  • Line Chart: For logs over time.
  • Data Table: For a structured log view.

Build a dashboard:

  1. Go to Dashboard > Create Dashboard.
  2. Add your visualizations.
  3. Save as "Java Application Log Monitoring".

Conclusion​


Congratulations! You've set up the ELK Stack on AWS EC2, integrated Filebeat for log shipping, and created a dashboard for real-time monitoring. This setup scales wellβ€”add more Filebeat instances for multi-app environments. For production, consider security enhancements like SSL, authentication, and backups.

If you encounter issues, check service logs with journalctl or Elastic's documentation. Happy logging!

β˜• Support My Work​


If you found this guide helpful, consider buying me a coffee to support my work!

Continue reading...
 


Join 𝕋𝕄𝕋 on Telegram
Channel PREVIEW:
You must log in or register to reply here.
Back
Top