beats

:tropical_fish: Beats - Lightweight shippers for Elasticsearch & Logstash

12,129

4,914

12,129

977

View on GitHub

Top Related Projects

prometheus

54,803

The Prometheus monitoring system and time series database.

telegraf

14,466

Agent for collecting, processing, aggregating, and writing metrics, logs, and other arbitrary data.

fluentd

12,814

Fluentd: Unified Logging Layer (project under CNCF)

logstash

14,172

Logstash - transport and process your logs, events, or other data

Quick Overview

Elastic Beats is a collection of lightweight, open-source data shippers for various types of operational data. These shippers, known as "Beats," are designed to send data from hundreds or thousands of machines and systems to Logstash or Elasticsearch. Beats are part of the Elastic Stack, working seamlessly with other Elastic products for data collection, visualization, and analysis.

Pros

Easy to deploy and configure, with minimal resource requirements
Extensible architecture allowing for custom beat development
Seamless integration with other Elastic Stack components
Wide range of pre-built beats for various data sources and use cases

Cons

Limited data processing capabilities compared to Logstash
Some beats may require additional setup or dependencies
Learning curve for developing custom beats
Performance can be impacted when collecting data from many sources simultaneously

Code Examples

Using Filebeat to ship log files:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/*.log

output.elasticsearch:
  hosts: ["localhost:9200"]

Configuring Metricbeat to collect system metrics:

metricbeat.modules:
- module: system
  metricsets:
    - cpu
    - memory
    - network
  period: 10s

output.elasticsearch:
  hosts: ["localhost:9200"]

Setting up Packetbeat to monitor network traffic:

packetbeat.interfaces.device: any

packetbeat.flows:
  timeout: 30s
  period: 10s

packetbeat.protocols:
- type: icmp
- type: dns
- type: http

output.elasticsearch:
  hosts: ["localhost:9200"]

Getting Started

To get started with Elastic Beats:

Download and install the desired Beat (e.g., Filebeat, Metricbeat, etc.) from the official Elastic website.
Configure the Beat by editing its configuration file (usually located in /etc/[beatname]/[beatname].yml).
Start the Beat service:
```
sudo service [beatname] start
```
Verify that data is being shipped to Elasticsearch or Logstash.
Use Kibana or other Elastic Stack components to visualize and analyze the collected data.

For more detailed instructions, refer to the official Elastic documentation for each specific Beat.

Competitor Comparisons

loki

23,307

Like Prometheus, but for logs.

Pros of Loki

Lightweight and efficient log aggregation system
Designed for easy integration with Grafana for visualization
Uses a label-based approach, making it more scalable for large datasets

Cons of Loki

Limited built-in data processing capabilities compared to Beats
Fewer out-of-the-box integrations with various data sources
Relatively newer project with a smaller ecosystem

Code Comparison

Loki (using promtail client):

scrape_configs:
  - job_name: system
    static_configs:
    - targets:
        - localhost
    labels:
      job: varlogs
      __path__: /var/log/*log

Beats (using Filebeat):

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/*.log
output.elasticsearch:
  hosts: ["localhost:9200"]

Both configurations demonstrate log collection from system files, but Beats offers more extensive options for data processing and output destinations out of the box.

prometheus

54,803

The Prometheus monitoring system and time series database.

Pros of Prometheus

More flexible query language (PromQL) for complex data analysis
Better suited for cloud-native and microservices architectures
Supports service discovery and dynamic environments

Cons of Prometheus

Less out-of-the-box data collection compared to Beats
Requires more setup and configuration for comprehensive monitoring
Limited long-term storage options without additional components

Code Comparison

Prometheus configuration (prometheus.yml):

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'node'
    static_configs:
      - targets: ['localhost:9100']

Beats configuration (metricbeat.yml):

metricbeat.modules:
- module: system
  metricsets: ["cpu", "memory", "network"]
  period: 10s

output.elasticsearch:
  hosts: ["localhost:9200"]

Both configurations demonstrate setting up basic metric collection, but Beats offers more pre-configured modules for various data sources, while Prometheus relies on exporters and custom scrape configurations.

telegraf

14,466

Agent for collecting, processing, aggregating, and writing metrics, logs, and other arbitrary data.

Pros of Telegraf

More extensive plugin ecosystem with support for a wider range of data sources and outputs
Simpler configuration and setup process
Better suited for collecting metrics from various systems and applications

Cons of Telegraf

Less tightly integrated with the Elastic Stack ecosystem
May require additional configuration for advanced log parsing and processing
Limited built-in visualizations compared to Beats' integration with Kibana

Code Comparison

Telegraf configuration example:

[[inputs.cpu]]
  percpu = true
  totalcpu = true
  collect_cpu_time = false
  report_active = false
[[outputs.influxdb]]
  urls = ["http://influxdb:8086"]

Beats configuration example:

metricbeat.modules:
- module: system
  metricsets: ["cpu"]
  enabled: true
  period: 10s
output.elasticsearch:
  hosts: ["elasticsearch:9200"]

Both projects use configuration files to define data collection and output settings. Telegraf uses TOML format, while Beats uses YAML. Telegraf's configuration is often more concise and straightforward, especially for simple use cases. Beats' configuration is more verbose but provides deeper integration with the Elastic Stack.

fluentd

12,814

Fluentd: Unified Logging Layer (project under CNCF)

Pros of Fluentd

More flexible and extensible plugin system
Better support for multiple output destinations
Lighter resource footprint, especially for memory usage

Cons of Fluentd

Steeper learning curve for configuration
Less out-of-the-box integrations compared to Beats
Can be slower in high-volume scenarios without proper tuning

Code Comparison

Fluentd configuration example:

<source>
  @type tail
  path /var/log/httpd-access.log
  tag apache.access
  <parse>
    @type apache2
  </parse>
</source>

Beats configuration example:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/httpd-access.log

Both Fluentd and Beats are popular log collection and forwarding tools, but they have different strengths. Fluentd offers more flexibility and customization options, while Beats provides a simpler setup and tighter integration with the Elastic Stack. The choice between them often depends on specific use cases and existing infrastructure.

logstash

14,172

Logstash - transport and process your logs, events, or other data

Pros of Logstash

More flexible and powerful data processing capabilities
Supports a wider range of input and output plugins
Better suited for complex data transformation and enrichment tasks

Cons of Logstash

Higher resource consumption and slower performance
Steeper learning curve and more complex configuration
Less suitable for lightweight data collection scenarios

Code Comparison

Logstash configuration example:

input {
  file {
    path => "/var/log/apache/access.log"
    start_position => "beginning"
  }
}

filter {
  grok {
    match => { "message" => "%{COMBINEDAPACHELOG}" }
  }
}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
  }
}

Beats configuration example (Filebeat):

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/apache/access.log

output.elasticsearch:
  hosts: ["localhost:9200"]

Beats offers a more straightforward configuration for simple data collection tasks, while Logstash provides more advanced processing capabilities at the cost of increased complexity. Beats is generally lighter on resources and faster, making it ideal for distributed data collection scenarios. Logstash excels in situations requiring extensive data manipulation and transformation before indexing.

Convert designs to code with AI

Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.

Try Visual Copilot

README

Beats - The Lightweight Shippers of the Elastic Stack

The Beats are lightweight data shippers, written in Go, that you install on your servers to capture all sorts of operational data (think of logs, metrics, or network packet data). The Beats send the operational data to Elasticsearch, either directly or via Logstash, so it can be visualized with Kibana.

By "lightweight", we mean that Beats have a small installation footprint, use limited system resources, and have no runtime dependencies.

This repository contains libbeat, our Go framework for creating Beats, and all the officially supported Beats:

Beat	Description
Auditbeat	Collect your Linux audit framework data and monitor the integrity of your files.
Filebeat	Tails and ships log files
Functionbeat	Read and ships events from serverless infrastructure.
Heartbeat	Ping remote services for availability
Metricbeat	Fetches sets of metrics from the operating system and services
Packetbeat	Monitors the network and applications by sniffing packets
Winlogbeat	Fetches and ships Windows Event logs
Osquerybeat	Runs Osquery and manages interraction with it.

In addition to the above Beats, which are officially supported by Elastic, the community has created a set of other Beats that make use of libbeat but live outside of this Github repository. We maintain a list of community Beats here.

Documentation and Getting Started

You can find the documentation and getting started guides for each of the Beats on the elastic.co site:

Documentation and Getting Started information for the Elastic Agent

You can find the documentation and getting started guides for the Elastic Agent on the elastic.co site

Getting Help

If you need help or hit an issue, please start by opening a topic on our discuss forums. Please note that we reserve GitHub tickets for confirmed bugs and enhancement requests.

Downloads

You can download pre-compiled Beats binaries, as well as packages for the supported platforms, from this page.

Contributing

We'd love working with you! You can help make the Beats better in many ways: report issues, help us reproduce issues, fix bugs, add functionality, or even create your own Beat.

Please start by reading our CONTRIBUTING file.

Building Beats from the Source

See our CONTRIBUTING file for information about setting up your dev environment to build Beats from the source.

Snapshots

For testing purposes, we generate snapshot builds that you can find here. Please be aware that these are built on top of main and are not meant for production.

CI

PR Comments

It is possible to trigger some jobs by putting a comment on a GitHub PR. (This service is only available for users affiliated with Elastic and not for open-source contributors.)

beats
- jenkins run the tests please or jenkins run tests or /test will kick off a default build.
- /test macos will kick off a default build with also the macos stages.
- /test <beat-name> will kick off the default build for the given PR in addition to the <beat-name> build itself.
- /test <beat-name> for macos will kick off a default build with also the macos stage for the <beat-name>.
apm-beats-update
- /run apm-beats-update
apm-beats-packaging
- /package or /packaging will kick of a build to generate the packages for beats.
apm-beats-tester
- /beats-tester will kick of a build to validate the generated packages.

PR Labels

It's possible to configure the build on a GitHub PR by labelling the PR with the below labels

<beat-name> to force the following builds to run the stages for the <beat-name>
macOS to force the following builds to run the macos stages.

Top Related Projects

Convert designs to code with AI

Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.

Try Visual Copilot