Skip to main content

📂 Installing and Configuring Filebeat 🎛️

Filebeat is a lightweight shipper for forwarding and centralizing log data. It is commonly used to send logs to Logstash for further processing before indexing them in Elasticsearch. In this guide, we will go through the installation and configuration of Filebeat to work with Logstash.

🚀 Step 1: Install Filebeat 🛠️​

First, install Filebeat using the following command:

sudo apt install filebeat

This will install Filebeat on your system.

⚙️ Step 2: Configure Filebeat​

Filebeat supports multiple output destinations. However, in this setup, we will use Logstash for further processing before sending logs to Elasticsearch.

1️⃣ Modify the Configuration File​

Open the Filebeat configuration file using:

sudo nano /etc/filebeat/filebeat.yml

🔹 Disable Direct Elasticsearch Output By default, Filebeat is configured to send logs directly to Elasticsearch. Since we are using Logstash, we need to disable it by commenting out the following lines:

#output.elasticsearch:
  # Array of hosts to connect to.
  #hosts: ["localhost:9200"]

🔹 Enable Logstash Output Next, configure Filebeat to send logs to Logstash by uncommenting the following lines:

output.logstash:
  # The Logstash hosts
  hosts: ["localhost:5044"]

This ensures that Filebeat sends logs to Logstash running on port 5044.

🌍 Can we change localhost to some other IP?​

Yes, instead of localhost, you can use a specific IP address (e.g., 192.168.1.100) when:

  • Logstash is running on another server in the network.
  • You have a distributed logging setup, and Filebeat is collecting logs from multiple remote machines.
  • The Elastic Stack components (Logstash, Elasticsearch, Kibana) are deployed across multiple nodes.

For example, if your Logstash server is running on 192.168.1.100, modify the Filebeat configuration:

output.logstash:
  hosts: ["192.168.1.100:5044"]

Similarly, for Elasticsearch:

output.elasticsearch:
  hosts: ["192.168.1.100:9200"]

🧩 Step 3: Enable Filebeat Modules​

Filebeat modules simplify log collection and parsing. We will use the system module, which collects logs from system services.

Enable the system module by running:

sudo filebeat modules enable system

Verify the enabled modules:

sudo filebeat modules list

By default, this module collects logs from syslog and authorization logs.

You can view the module’s configuration in:

cat /etc/filebeat/modules.d/system.yml

📦 What type of data does Filebeat send? 📡​

Filebeat collects and forwards different types of log data, including:

  • System logs (e.g., /var/log/syslog, /var/log/auth.log)
  • Application logs (e.g., Nginx, Apache, MySQL logs)
  • Container logs (e.g., Docker, Kubernetes logs)
  • Custom log files specified in the configuration

These logs contain valuable information such as timestamps, event types, error messages, user activities, and more, helping in monitoring and troubleshooting.

🏗️ Step 4: Load Filebeat Ingest Pipelines​

To process logs before sending them to Logstash, we need to set up ingest pipelines. Run the following command:

sudo filebeat setup --pipelines --modules system

Next, set up index management:

sudo filebeat setup --index-management -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=["localhost:9200"]'

📊 Step 5: Load Filebeat Dashboards into Kibana 📈​

Filebeat provides pre-built dashboards for Kibana to visualize log data. To load them, disable Logstash output temporarily and enable Elasticsearch output:

sudo filebeat setup -E output.logstash.enabled=false -E output.elasticsearch.hosts=['localhost:9200'] -E setup.kibana.host=localhost:5601

This step ensures that the dashboards are available in Kibana for visualizing logs.

🔄 Step 6: Start and Enable Filebeat Service​

Once the configuration is complete, start and enable Filebeat to run on system boot:

sudo systemctl start filebeat
sudo systemctl enable filebeat

Verify that Filebeat is running:

sudo systemctl status filebeat

🔍 Step 7: Verify Data in Elasticsearch​

To check if logs are reaching Elasticsearch, run:

curl -XGET 'http://192.168.1.100:9200/filebeat-*/_search?pretty'

If everything is configured correctly, you should see log entries in the output.

🎉 Conclusion​

You have successfully installed and configured Filebeat to collect logs and send them to Logstash for processing. With Elasticsearch and Kibana, you can now analyze and visualize your logs efficiently. 🚀 Happy Logging! 🎯