1 of 100

4.2 About

Concepts

Installation

Administration

Local testing

Data pipeline

Build from source code

You can download the most recent stable or development source code.

Stable

For production systems, it's strongly suggested that you get the latest stable release of the source code in either zip file or tarball file format from GitHub using the following link pattern:

https://github.com/fluent/fluent-bit/archive/refs/tags/v&lt;release version&gt;.tar.gz
https://github.com/fluent/fluent-bit/archive/refs/tags/v&lt;release version&gt;.zip

For example, for version 1.8.12 the link is: https://github.com/fluent/fluent-bit/archive/refs/tags/v1.8.12.tar.gz

Development

If you want to contribute to Fluent Bit, you should use the most recent code. You can get the development version from the Git repository:

The master branch is where the development of Fluent Bit happens. Development version users should expect issues when compiling or at run time.

Fluent Bit users are encouraged to help test every development version to ensure a stable release.

Next step

After downloading Fluent Bit, install it using one of the following methods:

$ sudo service fluent-bit status

● fluent-bit.service - Fluent Bit
   Loaded: loaded (/lib/systemd/system/fluent-bit.service; disabled; vendor preset: enabled)
   Active: active (running) since mié 2016-07-06 16:58:25 CST; 2h 45min ago
 Main PID: 6739 (fluent-bit)
    Tasks: 1
   Memory: 656.0K
      CPU: 1.393s
   CGroup: /system.slice/fluent-bit.service
           └─6739 /opt/fluent-bit/bin/fluent-bit -c /etc/fluent-bit/fluent-bit.conf
...

$ systemctl status fluent-bit

● fluent-bit.service - Fluent Bit
   Loaded: loaded (/lib/systemd/system/fluent-bit.service; disabled; vendor preset: enabled)
   Active: active (running) since mié 2016-07-06 16:58:25 CST; 2h 45min ago
 Main PID: 6739 (fluent-bit)
    Tasks: 1
   Memory: 656.0K
      CPU: 1.393s
   CGroup: /system.slice/fluent-bit.service
           └─6739 /opt/fluent-bit/bin/fluent-bit -c /etc/fluent-bit/fluent-bit.conf
...

service:
  flush: 1
  daemon: off
  log_level: info

pipeline:
  inputs:
    - name: cpu

[SERVICE]
  Flush     1
  Daemon    off
  Log_Level info

[INPUT]
  Name      cpu

[OUTPUT]
  Name      stdout
  Match     *

$ bin/fluent-bit

...
[0] cpu.local: [1539984752.000347547, {"cpu_p"=>0.750000, "user_p"=>0.500000, "system_p"=>0.250000, "cpu0.p_cpu"=>1.000000, "cpu0.p_user"=>1.000000, "cpu0.p_system"=>0.000000, "cpu1.p_cpu"=>0.000000, "cpu1.p_user"=>0.000000, "cpu1.p_system"=>0.000000, "cpu2.p_cpu"=>0.000000, "cpu2.p_user"=>0.000000, "cpu2.p_system"=>0.000000, "cpu3.p_cpu"=>1.000000, "cpu3.p_user"=>1.000000, "cpu3.p_system"=>0.000000}]

repo_gpgcheck

host

port

tls

tls_verify

shared_key

upstream_servers:
  - name: forward-balancing
    nodes:
      - name: node-1
        host: 127.0.0.1
        port: 43000

      - name: node-2
        host: 127.0.0.1
        port: 44000

      - name: node-3
        host: 127.0.0.1
        port: 45000
        tls: true
        tls_verify: false
        shared_key: secret

  - name: forward-balancing-2
    nodes:
      - name: node-A
        host: 192.168.1.10
        port: 50000

      - name: node-B
        host: 192.168.1.11
        port: 51000

$ systemctl status fluent-bit

● fluent-bit.service - Fluent Bit
   Loaded: loaded (/usr/lib/systemd/system/fluent-bit.service; disabled; vendor preset: disabled)
   Active: active (running) since Thu 2016-07-07 02:08:01 BST; 9s ago
 Main PID: 3820 (fluent-bit)
   CGroup: /system.slice/fluent-bit.service
           └─3820 /opt/fluent-bit/bin/fluent-bit -c etc/fluent-bit/fluent-bit.conf
...

To define custom parsers in the parsers section of a YAML configuration file, use the following syntax.

To define custom parsers in the multiline_parsers section of a YAML configuration file, use the following syntax:

20MB

Key concepts

Learn these key concepts to understand how Fluent Bit operates.

Before diving into Fluent Bit you might want to get acquainted with some of the key concepts of the service. This document provides an introduction to those concepts and common Fluent Bit terminology. Reading this document will help you gain a more general understanding of the following topics:

Event or Record
Filtering
Tag
Timestamp
Match
Structured Message

Events or records

Every incoming piece of data that belongs to a log or a metric that's retrieved by Fluent Bit is considered an Event or a Record.

As an example, consider the following content of a Syslog file:

It contains four lines that represent four independent Events.

An Event is comprised of:

timestamp
key/value metadata (v2.1.0 and greater)
payload

Event format

The Fluent Bit wire protocol represents an Event as a two-element array with a nested array as the first element:

where

TIMESTAMP is a timestamp in seconds as an integer or floating point value (not a string).
METADATA is an object containing event metadata, and might be empty.
MESSAGE is an object containing the event body.

Fluent Bit versions prior to v2.1.0 used:

to represent events. This format is still supported for reading input event streams.

Filtering

You might need to perform modifications on an event's content. The process to alter, append to, or drop Events is called .

Use filtering to:

Append specific information to the Event like an IP address or metadata.
Select a specific piece of the Event content.
Drop Events that match a certain pattern.

Tag

Every Event ingested by Fluent Bit is assigned a Tag. This tag is an internal string used in a later stage by the Router to decide which Filter or phase it must go through.

Most tags are assigned manually in the configuration. If a tag isn't specified, Fluent Bit assigns the name of the plugin instance where that Event was generated from.

The input plugin doesn't assign tags. This plugin speaks the Fluentd wire protocol called Forward where every Event already comes with a Tag associated. Fluent Bit will always use the incoming Tag set by the client.

A tagged record must always have a Matching rule. To learn more about Tags and Matches, see .

Timestamp

The timestamp represents the time an Event was created. Every Event contains an associated timestamps. All events have timestamps, and they're set by the input plugin or discovered through a data parsing process.

The timestamp is a numeric fractional integer in the format:

where:

_SECONDS_ is the number of seconds that have elapsed since the Unix epoch.
_NANOSECONDS_ is a fractional second or one thousand-millionth of a second.

Match

Fluent Bit lets you route your collected and processed Events to one or multiple destinations. A Match represents a rule to select Events where a Tag matches a defined rule.

To learn more about Tags and Matches, see .

Structured messages

Source events can have a structure. A structure defines a set of keys and values inside the Event message to implement faster operations on data modifications. Fluent Bit treats every Event message as a structured message.

Consider the following two messages:

No structured message
With a structured message

For performance reasons, Fluent Bit uses a binary serialization data format called .

Linux packages

Fluent Bit is available for a variety of Linux distributions and embedded Linux systems.

The most secure option is to create the repositories according to the instructions for your specific OS.

Single line install

An installation script is provided for use with most Linux targets. This will by default install the most recent version released.

curl https://raw.githubusercontent.com/fluent/fluent-bit/master/install.sh | sh

This is a helper and should always be validated prior to use.

GPG key updates

For the 1.9.0 and 1.8.15 releases and later, the GPG key . Ensure the new key is added.

The GPG Key fingerprint of the new key is:

The previous key is and might be required to install previous versions.

The GPG Key fingerprint of the old key is:

Refer to the to see which platforms are supported in each release.

Migration to Fluent Bit

For version 1.9 and later, td-agent-bit is a deprecated package and is removed after 1.9.9. The correct package name to use now is fluent-bit.

Environment variables

The env section lets you define environment variables directly within the configuration file. These variables can then be used to dynamically replace values throughout your configuration using the ${VARIABLE_NAME} syntax.

Variables set in this section can't be overridden by system environment variables.

Values set in the env section are case-sensitive. However, as a best practice, Fluent Bit recommends using uppercase names for environment variables. The following example defines two variables, FLUSH_INTERVAL and STDOUT_FMT, which can be accessed in the configuration using ${FLUSH_INTERVAL} and ${STDOUT_FMT}:

env:
  FLUSH_INTERVAL: 1
  STDOUT_FMT: 'json_lines'

service:
  flush: ${FLUSH_INTERVAL}
  log_level: info

pipeline:
  inputs:
    - name: random

  outputs:
    - name: stdout
      match: '*'
      format: ${STDOUT_FMT}

Predefined variables

Fluent Bit provides a set of predefined environment variables that can be used in your configuration:

Name

Description

External variables

In addition to variables defined in the configuration file or the predefined ones, Fluent Bit can access system environment variables set in the user space. These external variables can be referenced in the configuration using the same ${VARIABLE_NAME} pattern.

Variables set in the env section can't be overridden by system environment variables.

For example, to set the FLUSH_INTERVAL system environment variable to 2 and use it in your configuration:

In the configuration file, you can then access this value as follows:

This approach lets you manage and override configuration values using environment variables, providing flexibility in various deployment environments.

Includes

The includes section lets you specify additional YAML configuration files to be merged into the current configuration. These files are identified as a list of filenames and can include relative or absolute paths. If no absolute path is provided, the file is assumed to be located in a directory relative to the file that references it.

Use this section to organize complex configurations into smaller, manageable files and include them as needed.

Usage

The following example demonstrates how to include additional YAML files using relative path references. This is the file system path structure:

├── fluent-bit.yaml
├── inclusion-1.yaml
└── subdir
    └── inclusion-2.yaml

The content of fluent-bit.yaml:

Ensure that the included files are formatted correctly and contain valid YAML configurations for seamless integration.

If a path isn't specified as absolute, it will be treated as relative to the file that includes it.

Hot reload

Enable hot reload through SIGHUP signal or an HTTP endpoint

Fluent Bit supports the reloading feature when enabled in the configuration file or on the command line with -Y or --enable-hot-reload option.

Hot reloading is supported on Linux, macOS, and Windows operating systems.

Update the configuration

To get started with reloading over HTTP, enable the HTTP Server in the configuration file:

service:
  http_server: on
  http_listen: 0.0.0.0
  http_port: 2020
  hot_reload: on

How to reload

After updating the configuration, use one of the following methods to perform a hot reload:

HTTP

Use the following HTTP endpoints to perform a hot reload:

PUT /api/v2/reload
POST /api/v2/reload

For using curl to reload Fluent Bit, users must specify an empty request body as:

Obtain a count of hot reload using the HTTP endpoint:

GET /api/v2/reload

The endpoint returns hot_reload_count as follows:

The default value of the counter is 0.

Signal

Hot reloading can be used with SIGHUP.

SIGHUP signal isn't supported on Windows.

Confirm a reload

Use one of the following methods to confirm the reload occurred.

Performance tips

Fluent Bit is designed for high performance and minimal resource usage. Depending on your use case, you can optimize further using specific configuration options to achieve faster performance or reduce resource consumption.

Reading files with `tail`

The Tail input plugin is used to read data from files on the filesystem. By default, it uses a small memory buffer of 32KB per monitored file. While this is sufficient for most generic use cases and helps keep memory usage low when monitoring many files, there are scenarios where you might want to increase performance by using more memory.

If your files are typically larger than 32KB, consider increasing the buffer size to speed up file reading. For example, you can experiment with a buffer size of 128KB:

By increasing the buffer size, Fluent Bit will make fewer system calls (read(2)) to read the data, reducing CPU usage and improving performance.

Fluent Bit and SIMD for JSON encoding

The release of Fluent Bit v4.1.0 introduced new performance improvements for JSON encoding using Single Instruction, Multiple Data (SIMD). Plugins that convert logs from the Fluent Bit internal binary representation to JSON can now do so 2.5 times (read) faster. Powered by the .

Enabling SIMD support

Ensure that your Fluent Bit binary is built with SIMD support. This feature is available for architectures such as x86_64, amd64, aarch64, and arm64. As of now, SIMD is only enabled by default in Fluent Bit container images.

You can check if SIMD is enabled by looking for the following log entry when Fluent Bit starts:

Look for the simd entry, which will indicate the SIMD support type, such as SSE2, NEON, or none.

If your Fluent Bit binary wasn't built with SIMD enabled, and you are using a supported platform, you can build Fluent Bit from source using the CMake option -DFLB_SIMD=On.

Run input plugins in threaded mode

By default, most input plugins run in the same system thread than the main event loop, however by configuration you can instruct them to run in a separate thread which will allow you to take advantage of other CPU cores in your system.

To run an input plugin in threaded mode, add threaded: true as in the following example:

Run a logging pipeline locally

You can test logging pipelines locally to observe how they handles log messages. This guide explains how to use Docker Compose to run Fluent Bit and Elasticsearch locally, but you can use the same principles to test other plugins.

Create a configuration file

Start by creating one of the corresponding Fluent Bit configuration files to start testing.

pipeline:
  inputs:
    - name: dummy
      dummy: '{"top": {".dotted": "value"}}'
      
  outputs:       
    - name: es
      host

[INPUT]
  Name dummy
  Dummy {"top": {".dotted": "value"}}

[OUTPUT]
  Name es
  Host elasticsearch
  Replace_Dots On

Use Docker Compose

Use to run Fluent Bit (with the configuration file mounted) and Elasticsearch.

View indexed logs

To view indexed logs, run the following command:

Reset index

To reset your index, run the following command:

parsers:
  - name: custom_parser1
    format: json
    time_key: time
    time_format: '%Y-%m-%dT%H:%M:%S.%L'
    time_keep: on

  - name: custom_parser2
    format: regex
    regex: '^\<(?<pri>[0-9]{1,5})\>1 (?<time>[^ ]+) (?<host>[^ ]+) (?<ident>[^ ]+) (?<pid>[-0-9]+) (?<msgid>[^ ]+) (?<extradata>(\[(.*)\]|-)) (?<message>.+)$'
    time_key: time
    time_format: '%Y-%m-%dT%H:%M:%S.%L'
    time_keep: on
    types: pid:integer

$ bin/fluent-bit -c fluent-bit.conf

...
[0] cpu.local: [1491243925, {"cpu_p"=>1.750000, "user_p"=>1.750000, "system_p"=>0.000000, "cpu0.p_cpu"=>3.000000, "cpu0.p_user"=>2.000000, "cpu0.p_system"=>1.000000, "cpu1.p_cpu"=>0.000000, "cpu1.p_user"=>0.000000, "cpu1.p_system"=>0.000000, "cpu2.p_cpu"=>4.000000, "cpu2.p_user"=>4.000000, "cpu2.p_system"=>0.000000, "cpu3.p_cpu"=>1.000000, "cpu3.p_user"=>1.000000, "cpu3.p_system"=>0.000000}]

Build Flags =  JSMN_PARENT_LINKS JSMN_STRICT FLB_HAVE_TLS FLB_HAVE_SQLDB
FLB_HAVE_TRACE FLB_HAVE_FLUSH_LIBCO FLB_HAVE_VALGRIND FLB_HAVE_FORK
FLB_HAVE_PROXY_GO FLB_HAVE_JEMALLOC JEMALLOC_MANGLE FLB_HAVE_REGEX
FLB_HAVE_C_TLS FLB_HAVE_SETJMP FLB_HAVE_ACCEPT4 FLB_HAVE_INOTIFY

[fluent-bit]
  name = Fluent Bit
  baseurl = https://packages.fluentbit.io/amazonlinux/2/
  gpgcheck=1
  gpgkey=https://packages.fluentbit.io/fluentbit.key
  enabled=1

[fluent-bit]
  name = Fluent Bit
  baseurl = https://packages.fluentbit.io/amazonlinux/2023/
  gpgcheck=1
  gpgkey=https://packages.fluentbit.io/fluentbit.key
  enabled=1

PASS is the password when using basic authentication.

Jan 18 12:52:16 flb systemd[2222]: Starting GNOME Terminal Server
Jan 18 12:52:16 flb dbus-daemon[2243]: [session uid=1000 pid=2243] Successfully activated service 'org.gnome.Terminal'
Jan 18 12:52:16 flb systemd[2222]: Started GNOME Terminal Server.
Jan 18 12:52:16 flb gsd-media-keys[2640]: # watch_fast: "/org/gnome/terminal/legacy/" (establishing: 0, active: 0)

[2024/11/10 22:25:53] [ info] [fluent bit] version=3.2.0, commit=12cb22e0e9, pid=74359
[2024/11/10 22:25:53] [ info] [storage] ver=1.5.2, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2024/11/10 22:25:53] [ info] [simd    ] SSE2
[2024/11/10 22:25:53] [ info] [cmetrics] version=0.9.8
[2024/11/10 22:25:53] [ info] [ctraces ] version=0.5.7
[2024/11/10 22:25:53] [ info] [sp] stream processor started

docker-compose.yaml

version: "3.7"

services:
  fluent-bit:
    image: fluent/fluent-bit
    volumes:
      - ./fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf
    depends_on:
      - elasticsearch
  elasticsearch:
    image: elasticsearch:7.17.6
    ports:
      - "9200:9200"
    environment:
      - discovery.type=single-node

...
[0] disk.0: [1485590297, {"read_size"=>0, "write_size"=>0}]
[1] disk.0: [1485590298, {"read_size"=>0, "write_size"=>0}]
[2] disk.0: [1485590299, {"read_size"=>0, "write_size"=>0}]
[3] disk.0: [1485590300, {"read_size"=>0, "write_size"=>11997184}]
...

$ systemctl status fluent-bit

● fluent-bit.service - Fluent Bit
   Loaded: loaded (/usr/lib/systemd/system/fluent-bit.service; disabled; vendor preset: disabled)
   Active: active (running) since Thu 2016-07-07 02:08:01 BST; 9s ago
 Main PID: 3820 (fluent-bit)
   CGroup: /system.slice/fluent-bit.service
           └─3820 /opt/fluent-bit/bin/fluent-bit -c /etc/fluent-bit/fluent-bit.conf
...

$ sudo service fluent-bit status

● fluent-bit.service - Fluent Bit
   Loaded: loaded (/lib/systemd/system/fluent-bit.service; disabled; vendor preset: enabled)
   Active: active (running) since mié 2016-07-06 16:58:25 CST; 2h 45min ago
 Main PID: 6739 (fluent-bit)
    Tasks: 1
   Memory: 656.0K
      CPU: 1.393s
   CGroup: /system.slice/fluent-bit.service
           └─6739 /opt/fluent-bit/bin/fluent-bit -c /etc/fluent-bit/fluent-bit.conf
...

Record accessor syntax

A full feature set to access content of your records.

Fluent Bit works internally with structured records and it can be composed of an unlimited number of keys and values. Values can be anything like a number, string, array, or a map.

Having a way to select a specific part of the record is critical for certain core functionalities or plugins, this feature is called Record Accessor.

Consider record accessor to be a basic grammar to specify record content and other miscellaneous values.

Format

A record accessor rule starts with the character $. Use the structured content as an example. The following table describes how to access a record:

The following table describes some accessing rules and the expected returned value:

Format

Accessed Value

If the accessor key doesn't exist in the record like the last example $labels['undefined'], the operation is omitted, and no exception will occur.

Usage

The feature is enabled on a per plugin basis. Not all plugins enable this feature. As an example, consider a configuration that aims to filter records using that only matches where labels have a color blue:

The file content to process in test.log is the following:

When running Fluent Bit with the previous configuration, the output is:

Limitations of `record_accessor` templating

The Fluent Bit record_accessor library has a limitation in the characters that can separate template variables. Only dots and commas (. and ,) can come after a template variable. This is because the templating library must parse the template and determine the end of a variable.

The following templates are invalid because the template variables aren't separated by commas or dots:

$TaskID-$ECSContainerName
$TaskID/$ECSContainerName
$TaskID_$ECSContainerName

However, the following are valid:

$TaskID.$ECSContainerName
$TaskID.ecs_resource.$ECSContainerName
$TaskID.fooo.$ECSContainerName

And the following are valid since they only contain one template variable with nothing after it:

fooo$TaskID
fooo____$TaskID
fooo/bar$TaskID

Upstream servers

Fluent Bit output plugins aim to connect to external services to deliver logs over the network. Being able to connect to one node (host) is normal and enough for more of the use cases, but there are other scenarios where balancing across different nodes is required. The Upstream feature provides this capability.

An Upstream defines a set of nodes that will be targeted by an output plugin, by the nature of the implementation an output plugin must support the Upstream feature. The following plugin has Upstream support:

Forward

The current balancing mode implemented is round-robin.

Configuration

To define an Upstream you must create an specific configuration file that contains an UPSTREAM and one or multiple NODE sections. The following table describes the properties associated with each section. All properties are mandatory:

Section

Key

Description

Nodes and specific plugin configuration

A Node might contain additional configuration keys required by the plugin, to provide enough flexibility for the output plugin. A common use case is a Forward output where if TLS is enabled, it requires a shared key.

Nodes and TLS (Transport Layer Security)

In addition to the properties defined in the configuration table, the network operations against a defined node can optionally be done through the use of TLS for further encryption and certificates use.

The TLS options available are described in the section and can be added to the any Node section.

Configuration file example

The following example defines an Upstream called forward-balancing which aims to be used by Forward output plugin, it register three Nodes:

node-1: connects to 127.0.0.1:43000
node-2: connects to 127.0.0.1:44000
node-3: connects to 127.0.0.1:45000 using TLS without verification. It also defines a specific configuration option required by Forward output called shared_key.

Every Upstream definition must exists in its own configuration file in the file system. Adding multiple Upstream configurations in the same file or different files isn't allowed.

pipeline:
  inputs:
    - name: collectd
      listen: 0.0.0.0
      port: 25826
      typesdb: '/usr/share/collectd/types.db,/etc/collectd/custom.db'

  outputs:
    - name: stdout
      match: '*'

...
[0] kernel: [1463421823, {"priority"=>3, "sequence"=>1814, "sec"=>11706, "usec"=>732233, "msg"=>"ERROR @wl_cfg80211_get_station : Wrong Mac address, mac = 34:a8:4e:d3:40:ec profile =20:3a:07:9e:4a:ac"}]
[1] kernel: [1463421823, {"priority"=>3, "sequence"=>1815, "sec"=>11706, "usec"=>732300, "msg"=>"ERROR @wl_cfg80211_get_station : Wrong Mac address, mac = 34:a8:4e:d3:40:ec profile =20:3a:07:9e:4a:ac"}]
[2] kernel: [1463421829, {"priority"=>3, "sequence"=>1816, "sec"=>11712, "usec"=>729728, "msg"=>"ERROR @wl_cfg80211_get_station : Wrong Mac address, mac = 34:a8:4e:d3:40:ec profile =20:3a:07:9e:4a:ac"}]
[3] kernel: [1463421829, {"priority"=>3, "sequence"=>1817, "sec"=>11712, "usec"=>729802, "msg"=>"ERROR @wl_cfg80211_get_station : Wrong Mac address, mac = 34:a8:4e:d3:40:ec
...

$TaskIDfooo$ECSContainerName

fluent-bit

service:
  flush: 1
  log_level: info
    
pipeline:
  inputs:
    - name: kubernetes_events
      tag: k8s_events
      kube_url: https://kubernetes.default.svc
      
  outputs:
    - name: stdout
      match: '*'

[SERVICE]
  flush           1
  log_level       info

[INPUT]
  name            kubernetes_events
  tag             k8s_events
  kube_url        https://kubernetes.default.svc

[OUTPUT]
  name            stdout
  match           *

[SERVICE]
    flush        1
    log_level    info
    parsers_file parsers.conf

[INPUT]
    name      tail
    path      test.log
    parser    json

[FILTER]
    name      grep
    match     *
    regex     $labels['color'] ^blue$

[OUTPUT]
    name      stdout
    match     *
    format    json_lines

{"log": "message 1", "labels": {"color": "blue"}}
{"log": "message 2", "labels": {"color": "red"}}
{"log": "message 3", "labels": {"color": "green"}}
{"log": "message 4", "labels": {"color": "blue"}}

$ bin/fluent-bit -c fluent-bit.conf
Fluent Bit v1.x.x
* Copyright (C) 2019-2020 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2020/09/11 16:11:07] [ info] [engine] started (pid=1094177)
[2020/09/11 16:11:07] [ info] [storage] version=1.0.5, initializing...
[2020/09/11 16:11:07] [ info] [storage] in-memory
[2020/09/11 16:11:07] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2020/09/11 16:11:07] [ info] [sp] stream processor started
[2020/09/11 16:11:07] [ info] inotify_fs_add(): inode=55716713 watch_fd=1 name=test.log
{"date":1599862267.483684,"log":"message 1","labels":{"color":"blue"}}
{"date":1599862267.483692,"log":"message 4","labels":{"color":"blue"}}

[UPSTREAM]
  name       forward-balancing

[NODE]
  name       node-1
  host       127.0.0.1
  port       43000

[NODE]
  name       node-2
  host       127.0.0.1
  port       44000

[NODE]
  name       node-3
  host       127.0.0.1
  port       45000
  tls        on
  tls.verify off
  shared_key secret

...
[0] [1452185189, {"cpu_p"=>7.00, "user_p"=>5.00, "system_p"=>2.00, "cpu0.p_cpu"=>10.00, "cpu0.p_user"=>8.00, "cpu0.p_system"=>2.00, "cpu1.p_cpu"=>6.00, "cpu1.p_user"=>4.00, "cpu1.p_system"=>2.00}]
[1] [1452185190, {"cpu_p"=>6.50, "user_p"=>5.00, "system_p"=>1.50, "cpu0.p_cpu"=>6.00, "cpu0.p_user"=>5.00, "cpu0.p_system"=>1.00, "cpu1.p_cpu"=>7.00, "cpu1.p_user"=>5.00, "cpu1.p_system"=>2.00}]
[2] [1452185191, {"cpu_p"=>7.50, "user_p"=>5.00, "system_p"=>2.50, "cpu0.p_cpu"=>7.00, "cpu0.p_user"=>3.00, "cpu0.p_system"=>4.00, "cpu1.p_cpu"=>6.00, "cpu1.p_user"=>6.00, "cpu1.p_system"=>0.00}]
[3] [1452185192, {"cpu_p"=>4.50, "user_p"=>3.50, "system_p"=>1.00, "cpu0.p_cpu"=>6.00, "cpu0.p_user"=>5.00, "cpu0.p_system"=>1.00, "cpu1.p_cpu"=>5.00, "cpu1.p_user"=>3.00, "cpu1.p_system"=>2.00}]
...

...
[2025/12/02 08:33:54.689265000] [ info] [input:fluentbit_metrics:fluentbit_metrics.0] initializing
[2025/12/02 08:33:54.689272000] [ info] [input:fluentbit_metrics:fluentbit_metrics.0] storage_strategy='memory' (memory only)
[2025/12/02 08:33:54.689917000] [ info] [output:stdout:stdout.0] worker #0 started
[2025/12/02 08:33:54.690115000] [ info] [sp] stream processor started
[2025/12/02 08:33:54.690204000] [ info] [engine] Shutdown Grace Period=5, Shutdown Input Grace Period=2
2025-12-02T07:33:56.692855536Z fluentbit_uptime{hostname="XXXXX.local"} = 2
2025-12-02T07:33:54.687838528Z fluentbit_logger_logs_total{message_type="error"} = 0
2025-12-02T07:33:54.687838528Z fluentbit_logger_logs_total{message_type="warn"} = 0
2025-12-02T07:33:54.690212675Z fluentbit_logger_logs_total{message_type="info"} = 10
2025-12-02T07:33:54.687838528Z fluentbit_logger_logs_total{message_type="debug"} = 0
2025-12-02T07:33:54.687838528Z fluentbit_logger_logs_total{message_type="trace"} = 0
2025-12-02T07:33:54.689222850Z fluentbit_input_bytes_total{name="fluentbit_metrics.0"} = 0
2025-12-02T07:33:54.689222850Z fluentbit_input_records_total{name="fluentbit_metrics.0"} = 0
2025-12-02T07:33:54.689222850Z fluentbit_input_ring_buffer_writes_total{name="fluentbit_metrics.0"} = 0
2025-12-02T07:33:54.689222850Z fluentbit_input_ring_buffer_retries_total{name="fluentbit_metrics.0"} = 0
2025-12-02T07:33:54.689222850Z fluentbit_input_ring_buffer_retry_failures_total{name="fluentbit_metrics.0"} = 0
2025-12-02T07:33:56.692846827Z fluentbit_input_metrics_scrapes_total{name="fluentbit_metrics.0"} = 1
2025-12-02T07:33:54.689563930Z fluentbit_output_proc_records_total{name="stdout.0"} = 0
...

service:
  flush: 1
  log_level: info

pipeline:
  inputs:
    - name: fluentbit_metrics
      tag: internal_metrics
      scrape_interval: 2

  outputs:
    - name: prometheus_exporter
      match: internal_metrics
      host: 0.0.0.0
      port: 2021

# Fluent Bit Metrics + Prometheus Exporter
# -------------------------------------------
# The following example collects Fluent Bit metrics and exposes
# them through a Prometheus HTTP endpoint.
#
# After starting the service try it with:
#
# $ curl http://127.0.0.1:2021/metrics
#
[SERVICE]
  flush           1
  log_level       info

[INPUT]
  name            fluentbit_metrics
  tag             internal_metrics
  scrape_interval 2

[OUTPUT]
  name            prometheus_exporter
  match           internal_metrics
  host            0.0.0.0
  port            2021

$ fluent-bit -i health -p host=127.0.0.1 -p port=80 -o stdout

...
[0] health.0: [1624145988.305640385, {"alive"=>true}]
[1] health.0: [1624145989.305575360, {"alive"=>true}]
[2] health.0: [1624145990.306498573, {"alive"=>true}]
[3] health.0: [1624145991.305595498, {"alive"=>true}]
...

...
[0] memory: [[1751381087.225589224, {}], {"Mem.total"=>3986708, "Mem.used"=>560708, "Mem.free"=>3426000, "Swap.total"=>0, "Swap.used"=>0, "Swap.free"=>0}]
[0] memory: [[1751381088.228411537, {}], {"Mem.total"=>3986708, "Mem.used"=>560708, "Mem.free"=>3426000, "Swap.total"=>0, "Swap.used"=>0, "Swap.free"=>0}]
[0] memory: [[1751381089.225600084, {}], {"Mem.total"=>3986708, "Mem.used"=>561480, "Mem.free"=>3425228, "Swap.total"=>0, "Swap.used"=>0, "Swap.free"=>0}]
[0] memory: [[1751381090.228345064, {}], {"Mem.total"=>3986708, "Mem.used"=>561480, "Mem.free"=>3425228, "Swap.total"=>0, "Swap.used"=>0, "Swap.free"=>0}]
...

...
[0] memory: [[1751381087.225589224, {}], {"Mem.total"=>3986708, "Mem.used"=>560708, "Mem.free"=>3426000, "Swap.total"=>0, "Swap.used"=>0, "Swap.free"=>0, "proc_bytes"=>12349440, "proc_hr"=>"11.78M"}]
...

$ systemctl status fluent-bit

● fluent-bit.service - Fluent Bit
   Loaded: loaded (/usr/lib/systemd/system/fluent-bit.service; disabled; vendor preset: disabled)
   Active: active (running) since Thu 2016-07-07 02:08:01 BST; 9s ago
 Main PID: 3820 (fluent-bit)
   CGroup: /system.slice/fluent-bit.service
           └─3820 /opt/fluent-bit/bin/fluent-bit -c etc/fluent-bit/fluent-bit.conf
...

a [warn] [input] {input name or alias} paused (mem buf overlimit) log message.

pipeline:
  inputs:
    - name: random
      samples: -1
      interval_sec: 1
      interval_nsec: 0

  outputs:

[INPUT]
  Name          random
  Samples      -1
  Interval_Sec  1
  Interval_NSec 0

[OUTPUT]
  Name   stdout
  Match  *

CPack: Create package using productbuild
CPack: Install projects
CPack: - Run preinstall target for: fluent-bit
CPack: - Install project: fluent-bit []
CPack: -   Install component: binary
CPack: -   Install component: library
CPack: -   Install component: headers
CPack: -   Install component: headers-extra
CPack: Create package
CPack: -   Building component package: /Users/fluent-bit-builder/GitHub/fluent-bit/build/_CPack_Packages/Darwin/productbuild//Users/fluent-bit-builder/GitHub/fluent-bit/build/fluent-bit-1.9.2-apple/Contents/Packages/fluent-bit-1.9.2-apple-binary.pkg
CPack: -   Building component package: /Users/fluent-bit-builder/GitHub/fluent-bit/build/_CPack_Packages/Darwin/productbuild//Users/fluent-bit-builder/GitHub/fluent-bit/build/fluent-bit-1.9.2-apple/Contents/Packages/fluent-bit-1.9.2-apple-headers.pkg
CPack: -   Building component package: /Users/fluent-bit-builder/GitHub/fluent-bit/build/_CPack_Packages/Darwin/productbuild//Users/fluent-bit-builder/GitHub/fluent-bit/build/fluent-bit-1.9.2-apple/Contents/Packages/fluent-bit-1.9.2-apple-headers-extra.pkg
CPack: -   Building component package: /Users/fluent-bit-builder/GitHub/fluent-bit/build/_CPack_Packages/Darwin/productbuild//Users/fluent-bit-builder/GitHub/fluent-bit/build/fluent-bit-1.9.2-apple/Contents/Packages/fluent-bit-1.9.2-apple-library.pkg
CPack: - package: /Users/fluent-bit-builder/GitHub/fluent-bit/build/fluent-bit-1.9.2-apple.pkg generated.

pipeline:

  outputs:
    - name: http
      host: 192.168.5.6
      port: 8080
      retry_limit: false

    - name: es
      host: 192.168.5.20
      port: 9200
      logstash_format: on
      retry_limit: 5

[OUTPUT]
  Name        http
  Host        192.168.5.6
  Port        8080
  Retry_Limit False

[OUTPUT]
  Name            es
  Host            192.168.5.20
  Port            9200
  Logstash_Format On
  Retry_Limit     5

pipeline:
  inputs:
    - name: netif
      tag: netif
      interface: eth0
      interval_sec: 1
      interval_nsec: 0
      verbose: false
      test_at_init: false

  outputs:
    - name: stdout
      match: '*'

...
[0] netif.0: [1499524459.001698260, {"eth0.rx.bytes"=>89769869, "eth0.rx.packets"=>73357, "eth0.rx.errors"=>0, "eth0.tx.bytes"=>4256474, "eth0.tx.packets"=>24293, "eth0.tx.errors"=>0}]
[1] netif.0: [1499524460.002541885, {"eth0.rx.bytes"=>98, "eth0.rx.packets"=>1, "eth0.rx.errors"=>0, "eth0.tx.bytes"=>98, "eth0.tx.packets"=>1, "eth0.tx.errors"=>0}]
[2] netif.0: [1499524461.001142161, {"eth0.rx.bytes"=>98, "eth0.rx.packets"=>1, "eth0.rx.errors"=>0, "eth0.tx.bytes"=>98, "eth0.tx.packets"=>1, "eth0.tx.errors"=>0}]
[3] netif.0: [1499524462.002612971, {"eth0.rx.bytes"=>98, "eth0.rx.packets"=>1, "eth0.rx.errors"=>0, "eth0.tx.bytes"=>98, "eth0.tx.packets"=>1, "eth0.tx.errors"=>0}]
...

$ fluent-bit -i random -o stdout

...
[0] random.0: [1475893654, {"rand_value"=>1863375102915681408}]
[1] random.0: [1475893655, {"rand_value"=>425675645790600970}]
[2] random.0: [1475893656, {"rand_value"=>7580417447354808203}]
[3] random.0: [1475893657, {"rand_value"=>1501010137543905482}]
[4] random.0: [1475893658, {"rand_value"=>16238242822364375212}]
...

service:
  flush: 1
  daemon: off
  parsers_file: parsers.yaml
  log_level: info
  http_server: off
  http_listen: 0.0.0.0
  http_port: 2020

pipeline:
  inputs:
    - name: exec_wasi
      tag: exec.wasi.local
      wasi_path: /path/to/wasi/program.wasm
      # Note: run from the 'wasi_path' location.
      accessible_paths: /path/to/accessible
      parser: wasi

  outputs:
    - name: stdout
      match: '*'

[SERVICE]
  Flush        1
  Daemon       Off
  Parsers_File parsers.conf
  Log_Level    info
  HTTP_Server  Off
  HTTP_Listen  0.0.0.0
  HTTP_Port    2020

[INPUT]
  Name exec_wasi
  Tag  exec.wasi.local
  WASI_Path /path/to/wasi/program.wasm
  Accessible_Paths .,/path/to/accessible
  Parser wasi

[OUTPUT]
  Name  stdout
  Match *

pipeline:
    inputs:
        - name: tail
          path: /var/log/example.log
          parser: json

          processors:
              logs:
                  - name: record_modifier

    filters:
        - name: grep
          match: '*'
          regex: key pattern

    outputs:
        - name: stdout
          match: '*'

pipeline:
    inputs:
        - name: tail
          tag: syslog
          path: /var/log/syslog
    filters:
        - name: record_modifier
          match: syslog
          record:
              - powered_by calyptia
        - name: record_modifier
          match: syslog
          allowlist_key:
              - powered_by
              - message

pipeline:
  inputs:
    - name: prometheus_remote_write
      listen: 127.0.0.1
      port: 8080
      uri: /api/prom/push

  outputs

[INPUT]
  name prometheus_remote_write
  listen 127.0.0.1
  port 8080
  uri /api/prom/push

[OUTPUT]
  name stdout
  match *

pipeline:
  inputs:
    - name: prometheus_scrape
      host: 0.0.0.0
      port: 8201
      tag: vault
      metrics_path:

[INPUT]
  name prometheus_scrape
  host 0.0.0.0
  port 8201
  tag vault
  metrics_path /v1/sys/metrics?format=prometheus
  scrape_interval 10s

[OUTPUT]
  name stdout
  match *

Networking

Fluent Bit implements a unified networking interface that's exposed to components like plugins. This interface abstracts the complexity of general I/O and is fully configurable.

A common use case is when a component or plugin needs to connect with a service to send and receive data. There are many challenges to handle like unresponsive services, networking latency, or any kind of connectivity error. The networking interface aims to abstract and simplify the network I/O handling, minimize risks, and optimize performance.

Networking concepts

Fluent Bit uses the following networking concepts:

TCP connect timeout

Typically, creating a new TCP connection to a remote server is straightforward and takes a few milliseconds. However, there are cases where DNS resolving, a slow network, or incomplete TLS handshakes might create long delays, or incomplete connection statuses.

net.connect_timeout lets you configure the maximum time to wait for a connection to be established. This value already considers the TLS handshake process.
net.connect_timeout_log_error indicates if an error should be logged in case of connect timeout. If disabled, the timeout is logged as a debug level message.

TCP source address

On environments with multiple network interfaces, you can choose which interface to use for Fluent Bit data that will flow through the network.

Use net.source_address to specify which network address to use for a TCP connection and data flow.

Connection keepalive

A connection keepalive refers to the ability of a client to keep the TCP connection open in a persistent way. This feature offers many benefits in terms of performance because communication channels are always established beforehand.

Any component that uses TCP channels like HTTP or , can take use feature. For configuration purposes use the net.keepalive property.

Connection keepalive idle timeout

If a connection keepalive is enabled, there might be scenarios where the connection can be unused for long periods of time. Unused connections can be removed. To control how long a keepalive connection can be idle, Fluent Bit uses a configuration property called net.keepalive_idle_timeout.

DNS mode

The global dns.mode value issues DNS requests using the specified protocol, either TCP or UDP. If a transport layer protocol is specified, plugins that configure the net.dns.mode setting override the global setting.

Maximum connections per worker

For optimal performance, Fluent Bit tries to deliver data quickly and create TCP connections on-demand and in keepalive mode. In highly scalable environments, you might limit how many connections are created in parallel.

Use the net.max_worker_connections property in the output plugin section to set the maximum number of allowed connections. This property acts at the worker level. For example, if you have five workers and net.max_worker_connections is set to 10, a maximum of 50 connections is allowed. If the limit is reached, the output plugin issues a retry.

Listener backlog

When Fluent Bit listens for incoming connections (for example, in input plugins like HTTP, TCP, OpenTelemetry, Forward, and Syslog), the operating system maintains a queue of pending connections. The net.backlog option controls the maximum number of pending connections that can be queued before new connection attempts are refused. Increasing this value can help Fluent Bit handle bursts of incoming connections more gracefully. The default value is 128.

On Linux, the effective backlog value might be capped by the kernel parameter net.core.somaxconn. If you need to allow a greater number of pending connections, you can increase this system setting.

Configuration options

The following table describes the network configuration properties available and their usage in optimizing performance or adjusting configuration needs for plugins that rely on networking I/O:

Property

Description

Default

Example

This example sends five random messages through a TCP output connection. The remote side uses the nc (netcat) utility to see the data.

Use the following configuration snippet of your choice in a corresponding file named fluent-bit.yaml or fluent-bit.conf:

In another terminal, start nc and make it listen for messages on TCP port 9090:

Start Fluent Bit with the configuration file you defined previously to see data flowing to netcat:

If the net.keepalive option isn't enabled, Fluent Bit closes the TCP connection and netcat quits.

After the five records arrive, the connection idles. After 10 seconds, the connection closes due to net.keepalive_idle_timeout.

eBPF

This plugin is experimental and might be unstable. Use it in development or testing environments only. Its features and behavior are subject to change.

The in_ebpf input plugin uses eBPF (extended Berkeley Packet Filter) to capture low-level system events. This plugin lets Fluent Bit monitor kernel-level activities such as process executions, file accesses, memory allocations, network connections, and signal handling. It provides valuable insights into system behavior for debugging, monitoring, and security analysis.

The in_ebpf plugin leverages eBPF to trace kernel events in real-time. By specifying trace points, users can collect targeted system-level metrics and events, giving visibility into operating system interactions and performance characteristics.

Configuration parameters

The plugin supports the following configuration parameters:

Key

Description

Default

System dependencies

To enable in_ebpf, ensure the following dependencies are installed on your system:

Kernel version: 4.18 or greater, with eBPF support enabled.
Required packages:
- bpftool: Used to manage and debug eBPF programs.

Installing dependencies on Ubuntu

Building Fluent Bit with `in_ebpf`

To enable the in_ebpf plugin, follow these steps to build Fluent Bit from source:

Clone the Fluent Bit repository:
Configure the build with in_ebpf:
Create a build directory and run cmake with the -DFLB_IN_EBPF=On flag to enable the in_ebpf plugin:
Compile the source:

Configuration example

Here's a basic example of how to configure the plugin:

The configuration enables tracing for:

Signal handling events (trace_signal)
Memory allocation events (trace_malloc)
Network bind operations (trace_bind)

You can enable multiple traces by adding multiple Trace directives in your configuration. Full list of existing traces can be seen here:

Output fields

Each trace produces records with common fields and trace-specific fields.

Common fields

All traces include the following fields:

Field

Description

Signal trace fields

The trace_signal trace includes these additional fields:

Field

Description

Memory trace fields

The trace_malloc trace includes these additional fields:

Field

Description

Bind trace fields

The trace_bind trace includes these additional fields:

Field

Description

Exec

The Exec input plugin lets you execute external programs and collects event logs.

This plugin invokes commands using a shell. Its inputs are subject to shell metacharacter substitution. Careless use of untrusted input in command arguments could lead to malicious command execution.

Container support

This plugin needs a functional /bin/sh and won't function in all the distro-less production images.

The debug images use the same binaries so even though they have a shell, there is no support for this plugin as it's compiled out.

Configuration parameters

The plugin supports the following configuration parameters:

Key

Description

Default

Get started

You can run the plugin from the command line or through the configuration file:

Command line

The following example will read events from the output of ls.

which should return something like the following:

Configuration file

In your main configuration file append the following:

Use as a command wrapper

To use Fluent Bit with the exec plugin to wrap another command, use the exit_after_oneshot and propagate_exit_code options:

Fluent Bit will output:

then exits with exit code 1.

Translation of command exit codes to Fluent Bit exit code follows . Like with a shell, there is no way to differentiate between the command exiting on a signal and the shell exiting on a signal. Similarly, there is no way to differentiate between normal exits with codes greater than 125 and abnormal or signal exits reported by Fluent Bit or the shell. Wrapped commands should use exit codes between 0 and 125 inclusive to allow reliable identification of normal exit. If the command is a pipeline, the exit code will be the exit code of the last command in the pipeline unless overridden by shell options.

Parsing command output

By default, the exec plugin emits one message per command output line, with a single field exec containing the full message. Use the parser option to specify the name of a parser configuration to use to process the command input.

Security concerns

Take great care with shell quoting and escaping when wrapping commands.

A script like the following can ruin your day if someone passes it the argument $(rm -rf /my/important/files; echo "deleted your stuff!")'

The previous script would be safer if written with:

It's generally best to avoid dynamically generating the command or handling untrusted arguments.

$ fluent-bit -i proc -p proc_name=fluent-bit -o stdout

...
[0] proc.0: [1485780297, {"alive"=>true, "proc_name"=>"fluent-bit", "pid"=>10964, "mem.VmPeak"=>14740000, "mem.VmSize"=>14740000, "mem.VmLck"=>0, "mem.VmHWM"=>1120000, "mem.VmRSS"=>1120000, "mem.VmData"=>2276000, "mem.VmStk"=>88000, "mem.VmExe"=>1768000, "mem.VmLib"=>2328000, "mem.VmPTE"=>68000, "mem.VmSwap"=>0, "fd"=>18}]
[1] proc.0: [1485780298, {"alive"=>true, "proc_name"=>"fluent-bit", "pid"=>10964, "mem.VmPeak"=>14740000, "mem.VmSize"=>14740000, "mem.VmLck"=>0, "mem.VmHWM"=>1148000, "mem.VmRSS"=>1148000, "mem.VmData"=>2276000, "mem.VmStk"=>88000, "mem.VmExe"=>1768000, "mem.VmLib"=>2328000, "mem.VmPTE"=>68000, "mem.VmSwap"=>0, "fd"=>18}]
[2] proc.0: [1485780299, {"alive"=>true, "proc_name"=>"fluent-bit", "pid"=>10964, "mem.VmPeak"=>14740000, "mem.VmSize"=>14740000, "mem.VmLck"=>0, "mem.VmHWM"=>1152000, "mem.VmRSS"=>1148000, "mem.VmData"=>2276000, "mem.VmStk"=>88000, "mem.VmExe"=>1768000, "mem.VmLib"=>2328000, "mem.VmPTE"=>68000, "mem.VmSwap"=>0, "fd"=>18}]
[3] proc.0: [1485780300, {"alive"=>true, "proc_name"=>"fluent-bit", "pid"=>10964, "mem.VmPeak"=>14740000, "mem.VmSize"=>14740000, "mem.VmLck"=>0, "mem.VmHWM"=>1152000, "mem.VmRSS"=>1148000, "mem.VmData"=>2276000, "mem.VmStk"=>88000, "mem.VmExe"=>1768000, "mem.VmLib"=>2328000, "mem.VmPTE"=>68000, "mem.VmSwap"=>0, "fd"=>18}]
...

service:
    flush: 1
    log_level: info
    parsers_file: parsers.yaml

pipeline:
    inputs:
        - name: tail
          path: data.log
          parser: json
          exit_on_eof: on

    # First 'expect' filter to validate that our data was structured properly
    filters:
        - name: expect
          match: '*'
          key_exists:
            - color
            - $label['name']
          action: exit

    outputs:
        - name: stdout
          match: '*'

[SERVICE]
    flush        1
    log_level    info
    parsers_file parsers.conf

[INPUT]
    name        tail
    path        ./data.log
    parser      json
    exit_on_eof on

# First 'expect' filter to validate that our data was structured properly
[FILTER]
    name        expect
    match       *
    key_exists  color
    key_exists  $label['name']
    action      exit

[OUTPUT]
    name        stdout
    match       *

service:
    flush: 1
    log_level: info
    parsers_file: parsers.yaml

pipeline:
    inputs:
        - name: tail
          path: data.log
          parser: json
          exit_on_eof: on

    # First 'expect' filter to validate that our data was structured properly
    filters:
        - name: expect
          match: '*'
          key_exists:
            - color
            - $label['name']
          action: exit

        # Match records that only contains map 'label' with key 'name' = 'abc'
        - name: grep
          match: '*'
          regex: "$label['name'] ^abc$"

        # Check that every record contains 'label' with a non-null value
        - name: expect
          match: '*'
          key_val_eq: $label['name'] abc
          action: exit

        # Append a new key to the record using an environment variable
        - name: record_modifier
          match: '*'
          record: hostname ${HOSTNAME}

        # Check that every record contains 'hostname' key
        - name: expect
          match: '*'
          key_exists: hostname
          action: exit

    outputs:
        - name: stdout
          match: '*'

[SERVICE]
    flush        1
    log_level    info
    parsers_file parsers.conf

[INPUT]
    name         tail
    path         ./data.log
    parser       json
    exit_on_eof  on

# First 'expect' filter to validate that our data was structured properly
[FILTER]
    name       expect
    match      *
    key_exists color
    key_exists label
    action     exit

# Match records that only contains map 'label' with key 'name' = 'abc'
[FILTER]
    name       grep
    match      *
    regex      $label['name'] ^abc$

# Check that every record contains 'label' with a non-null value
[FILTER]
    name       expect
    match      *
    key_val_eq $label['name'] abc
    action     exit

# Append a new key to the record using an environment variable
[FILTER]
    name       record_modifier
    match      *
    record     hostname ${HOSTNAME}

# Check that every record contains 'hostname' key
[FILTER]
    name       expect
    match      *
    key_exists hostname
    action     exit

[OUTPUT]
    name       stdout
    match      *

pipeline:
  inputs:
    - name: head
      tag: head.cpu
      file: /proc/cpuinfo
      lines: 8
      split_line: true

  filters:
    - name: record_modifier
      match: '*'
      whitelist_key: line7

  outputs:
    - name: stdout
      match: '*'

[INPUT]
  Name           head
  Tag            head.cpu
  File           /proc/cpuinfo
  Lines          8
  Split_Line     true
  # {"line0":"processor    : 0", "line1":"vendor_id    : GenuineIntel" ...}

[FILTER]
  Name           record_modifier
  Match          *
  Whitelist_key  line7

[OUTPUT]
  Name   stdout
  Match  *

Fluent Bit v1.x.x
* Copyright (C) 2019-2020 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2016/05/17 21:53:54] [ info] starting engine
[0] uptime: [1463543634, {"head"=>"133517.70 194870.97"}]
[1] uptime: [1463543635, {"head"=>"133518.70 194872.85"}]
[2] uptime: [1463543636, {"head"=>"133519.70 194876.63"}]
[3] uptime: [1463543637, {"head"=>"133520.70 194879.72"}]

processor    : 0
vendor_id    : GenuineIntel
cpu family   : 6
model        : 42
model name   : Intel(R) Core(TM) i7-2640M CPU @ 2.80GHz
stepping     : 7
microcode    : 41
cpu MHz      : 2791.009
cache size   : 4096 KB
physical id  : 0
siblings     : 1

Fluent Bit v1.x.x
* Copyright (C) 2019-2020 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2017/06/26 22:38:24] [ info] [engine] started
[0] head.cpu: [1498484305.000279805, {"line7"=>"cpu MHz        : 2791.009"}]
[1] head.cpu: [1498484306.011680137, {"line7"=>"cpu MHz        : 2791.009"}]
[2] head.cpu: [1498484307.010042482, {"line7"=>"cpu MHz        : 2791.009"}]
[3] head.cpu: [1498484308.008447978, {"line7"=>"cpu MHz        : 2791.009"}]

# HELP fluentbit_input_bytes_total Number of input bytes.
# TYPE fluentbit_input_bytes_total counter
fluentbit_input_bytes_total{name="podman_metrics.0"} 0
# HELP fluentbit_input_records_total Number of input records.
# TYPE fluentbit_input_records_total counter
fluentbit_input_records_total{name="podman_metrics.0"} 0
# HELP container_memory_usage_bytes Container memory usage in bytes
# TYPE container_memory_usage_bytes counter
container_memory_usage_bytes{id="858319c39f3f52cd44aa91a520aafb84ded3bc4b4a1e04130ccf87043149bbbf",name="blissful_wescoff",image="docker.io/library/ubuntu:latest"} 884736
# HELP container_cpu_user_seconds_total Container cpu usage in seconds in user mode
# TYPE container_cpu_user_seconds_total counter
container_cpu_user_seconds_total{id="858319c39f3f52cd44aa91a520aafb84ded3bc4b4a1e04130ccf87043149bbbf",name="blissful_wescoff",image="docker.io/library/ubuntu:latest"} 0
# HELP container_cpu_usage_seconds_total Container cpu usage in seconds
# TYPE container_cpu_usage_seconds_total counter
container_cpu_usage_seconds_total{id="858319c39f3f52cd44aa91a520aafb84ded3bc4b4a1e04130ccf87043149bbbf",name="blissful_wescoff",image="docker.io/library/ubuntu:latest"} 0
# HELP container_network_receive_bytes_total Network received bytes
# TYPE container_network_receive_bytes_total counter
container_network_receive_bytes_total{id="858319c39f3f52cd44aa91a520aafb84ded3bc4b4a1e04130ccf87043149bbbf",name="blissful_wescoff",image="docker.io/library/ubuntu:latest",interface="eth0"} 8515
# HELP container_network_receive_errors_total Network received errors
# TYPE container_network_receive_errors_total counter
container_network_receive_errors_total{id="858319c39f3f52cd44aa91a520aafb84ded3bc4b4a1e04130ccf87043149bbbf",name="blissful_wescoff",image="docker.io/library/ubuntu:latest",interface="eth0"} 0
# HELP container_network_transmit_bytes_total Network transmitted bytes
# TYPE container_network_transmit_bytes_total counter
container_network_transmit_bytes_total{id="858319c39f3f52cd44aa91a520aafb84ded3bc4b4a1e04130ccf87043149bbbf",name="blissful_wescoff",image="docker.io/library/ubuntu:latest",interface="eth0"} 962
# HELP container_network_transmit_errors_total Network transmitted errors
# TYPE container_network_transmit_errors_total counter
container_network_transmit_errors_total{id="858319c39f3f52cd44aa91a520aafb84ded3bc4b4a1e04130ccf87043149bbbf",name="blissful_wescoff",image="docker.io/library/ubuntu:latest",interface="eth0"} 0
# HELP fluentbit_input_storage_overlimit Is the input memory usage overlimit ?.
# TYPE fluentbit_input_storage_overlimit gauge
fluentbit_input_storage_overlimit{name="podman_metrics.0"} 0
# HELP fluentbit_input_storage_memory_bytes Memory bytes used by the chunks.
# TYPE fluentbit_input_storage_memory_bytes gauge
fluentbit_input_storage_memory_bytes{name="podman_metrics.0"} 0
# HELP fluentbit_input_storage_chunks Total number of chunks.
# TYPE fluentbit_input_storage_chunks gauge
fluentbit_input_storage_chunks{name="podman_metrics.0"} 0
# HELP fluentbit_input_storage_chunks_up Total number of chunks up in memory.
# TYPE fluentbit_input_storage_chunks_up gauge
fluentbit_input_storage_chunks_up{name="podman_metrics.0"} 0
# HELP fluentbit_input_storage_chunks_down Total number of chunks down.
# TYPE fluentbit_input_storage_chunks_down gauge
fluentbit_input_storage_chunks_down{name="podman_metrics.0"} 0
# HELP fluentbit_input_storage_chunks_busy Total number of chunks in a busy state.
# TYPE fluentbit_input_storage_chunks_busy gauge
fluentbit_input_storage_chunks_busy{name="podman_metrics.0"} 0
# HELP fluentbit_input_storage_chunks_busy_bytes Total number of bytes used by chunks in a busy state.
# TYPE fluentbit_input_storage_chunks_busy_bytes gauge
fluentbit_input_storage_chunks_busy_bytes{name="podman_metrics.0"} 0

pipeline:
  inputs:
    - name: prometheus_remote_write
      listen: 127.0.0.1
      port: 8080
      uri: /api/prom/push
      tls: on
      tls.crt_file: /path/to/certificate.crt
      tls.key_file: /path/to/certificate.key

...
2022-03-26T23:01:29.836663788Z go_memstats_alloc_bytes_total = 31891336
2022-03-26T23:01:29.836663788Z go_memstats_frees_total = 313264
2022-03-26T23:01:29.836663788Z go_memstats_lookups_total = 0
2022-03-26T23:01:29.836663788Z go_memstats_mallocs_total = 378992
2022-03-26T23:01:29.836663788Z process_cpu_seconds_total = 1.6200000000000001
2022-03-26T23:01:29.836663788Z go_goroutines = 19
2022-03-26T23:01:29.836663788Z go_info{version="go1.17.7"} = 1
2022-03-26T23:01:29.836663788Z go_memstats_alloc_bytes = 12547800
2022-03-26T23:01:29.836663788Z go_memstats_buck_hash_sys_bytes = 1468900
2022-03-26T23:01:29.836663788Z go_memstats_gc_cpu_fraction = 8.1509688352783453e-06
2022-03-26T23:01:29.836663788Z go_memstats_gc_sys_bytes = 5875576
2022-03-26T23:01:29.836663788Z go_memstats_heap_alloc_bytes = 12547800
2022-03-26T23:01:29.836663788Z go_memstats_heap_idle_bytes = 2220032
2022-03-26T23:01:29.836663788Z go_memstats_heap_inuse_bytes = 14000128
2022-03-26T23:01:29.836663788Z go_memstats_heap_objects = 65728
2022-03-26T23:01:29.836663788Z go_memstats_heap_released_bytes = 2187264
2022-03-26T23:01:29.836663788Z go_memstats_heap_sys_bytes = 16220160
2022-03-26T23:01:29.836663788Z go_memstats_last_gc_time_seconds = 1648335593.2483871
2022-03-26T23:01:29.836663788Z go_memstats_mcache_inuse_bytes = 2400
2022-03-26T23:01:29.836663788Z go_memstats_mcache_sys_bytes = 16384
2022-03-26T23:01:29.836663788Z go_memstats_mspan_inuse_bytes = 150280
2022-03-26T23:01:29.836663788Z go_memstats_mspan_sys_bytes = 163840
2022-03-26T23:01:29.836663788Z go_memstats_next_gc_bytes = 16586496
2022-03-26T23:01:29.836663788Z go_memstats_other_sys_bytes = 422572
2022-03-26T23:01:29.836663788Z go_memstats_stack_inuse_bytes = 557056
2022-03-26T23:01:29.836663788Z go_memstats_stack_sys_bytes = 557056
2022-03-26T23:01:29.836663788Z go_memstats_sys_bytes = 24724488
2022-03-26T23:01:29.836663788Z go_threads = 8
2022-03-26T23:01:29.836663788Z process_max_fds = 65536
2022-03-26T23:01:29.836663788Z process_open_fds = 12
2022-03-26T23:01:29.836663788Z process_resident_memory_bytes = 200638464
2022-03-26T23:01:29.836663788Z process_start_time_seconds = 1648333791.45
2022-03-26T23:01:29.836663788Z process_virtual_memory_bytes = 865849344
2022-03-26T23:01:29.836663788Z process_virtual_memory_max_bytes = 1.8446744073709552e+19
2022-03-26T23:01:29.836663788Z vault_runtime_alloc_bytes = 12482136
2022-03-26T23:01:29.836663788Z vault_runtime_free_count = 313256
2022-03-26T23:01:29.836663788Z vault_runtime_heap_objects = 65465
2022-03-26T23:01:29.836663788Z vault_runtime_malloc_count = 378721
2022-03-26T23:01:29.836663788Z vault_runtime_num_goroutines = 12
2022-03-26T23:01:29.836663788Z vault_runtime_sys_bytes = 24724488
2022-03-26T23:01:29.836663788Z vault_runtime_total_gc_pause_ns = 1917611
2022-03-26T23:01:29.836663788Z vault_runtime_total_gc_runs = 19
...

libbpf-dev

libbpf

service:
  flush: 1
  log_level: info

pipeline:
  inputs:
    - name:  random
      samples: 5

  outputs:
    - name: tcp
      match: '*'
      host: 127.0.0.1
      port: 9090
      format: json_lines
      # Networking Setup
      net.dns.mode: TCP
      net.connect_timeout: 5s
      net.source_address: 127.0.0.1
      net.keepalive: true
      net.keepalive_idle_timeout: 10s

[SERVICE]
  flush     1
  log_level info

[INPUT]
  name      random
  samples   5

[OUTPUT]
  name      tcp
  match     *
  host      127.0.0.1
  port      9090
  format    json_lines
  # Networking Setup
  net.dns.mode                TCP
  net.connect_timeout         5s
  net.source_address          127.0.0.1
  net.keepalive               true
  net.keepalive_idle_timeout  10s

$ nc -l 9090

{"date":1587769732.572266,"rand_value":9704012962543047466}
{"date":1587769733.572354,"rand_value":7609018546050096989}
{"date":1587769734.572388,"rand_value":17035865539257638950}
{"date":1587769735.572419,"rand_value":17086151440182975160}
{"date":1587769736.572277,"rand_value":527581343064950185}

# Process Exporter Metrics + Prometheus Exporter
# -------------------------------------------
# The following example collect host metrics on Linux and expose
# them through a Prometheus HTTP endpoint.
#
# After starting the service try it with:
#
# $ curl http://127.0.0.1:2021/metrics
#
service:
  flush: 1
  log_level: info

pipeline:
  inputs:
    - name: process_exporter_metrics
      tag:  process_metrics
      scrape_interval: 2

  outputs:
    - name: prometheus_exporter
      match: process_metrics
      host: 0.0.0.0
      port: 2021

# Process Exporter Metrics + Prometheus Exporter
# -------------------------------------------
# The following example collect host metrics on Linux and expose
# them through a Prometheus HTTP endpoint.
#
# After starting the service try it with:
#
# $ curl http://127.0.0.1:2021/metrics
#
[SERVICE]
  flush           1
  log_level       info

[INPUT]
  name            process_exporter_metrics
  tag             process_metrics
  scrape_interval 2

[OUTPUT]
  name            prometheus_exporter
  match           process_metrics
  host            0.0.0.0
  port            2021

docker run -ti -v /proc:/host/proc:ro \
            -p 2021:2021        \
            fluent/fluent-bit:2.2 \
            /fluent-bit/bin/fluent-bit \
            -i process_exporter_metrics \
            -p path.procfs=/host/proc  \
            -o prometheus_exporter \
            -f 1

Kafka

The Kafka input plugin enables Fluent Bit to consume messages directly from one or more Apache Kafka topics. By subscribing to specified topics, this plugin efficiently collects and forwards Kafka messages for further processing within your Fluent Bit pipeline.

Starting with version 4.0.4, the Kafka input plugin supports authentication with AWS MSK IAM, enabling integration with Amazon MSK (Managed Streaming for Apache Kafka) clusters that require IAM-based access.

This plugin uses the official librdkafka C library as a built-in dependency.

Configuration parameters

Key

Description

Default

Get started

To subscribe to or collect messages from Apache Kafka, run the plugin from the command line or through the configuration file as shown in the following examples.

Command line

The Kafka plugin can read parameters through the -p argument (property):

Configuration file (recommended)

In your main configuration file append the following:

Example of using Kafka input and output plugins

The Fluent Bit source repository contains a full example of using Fluent Bit to process Kafka records:

The previous example will connect to the broker listening on kafka-broker:9092 and subscribe to the fb-source topic, polling for new messages every 100 milliseconds.

Since the payload will be in JSON format, the plugin is configured to parse the payload with format json.

Every message received is then processed with kafka.lua and sent back to the fb-sink topic of the same broker.

The example can be executed locally with make start in the examples/kafka_filter directory (docker/compose is used).

AWS MSK IAM authentication

Fluent Bit v4.0.4 and later supports authentication to Amazon MSK (Managed Streaming for Apache Kafka) clusters using AWS IAM. This lets you securely connect to MSK brokers with AWS credentials, leveraging IAM roles and policies for access control.

Build requirements

If you are compiling Fluent Bit from source, ensure the following requirements are met to enable AWS MSK IAM support:

The packages libsasl2 and libsasl2-dev must be installed on your build environment.

Runtime requirements

Network Access: Fluent Bit must be able to reach your MSK broker endpoints (AWS VPC setup).
AWS Credentials: Provide these AWS credentials using any supported AWS method. These credentials are discovered by default when aws_msk_iam flag is enabled.
- IAM roles (recommended for EC2, ECS, or EKS)

Configuration parameters [#config-aws]

Property

Description

Required

Configuration example

Example AWS IAM policy

IAM policies and permissions can be complex and might vary depending on your organization's security requirements. If you are unsure about the correct permissions or best practices, consult your AWS administrator or an AWS expert who is familiar with MSK and IAM security.

The AWS credentials used by Fluent Bit must have permission to connect to your MSK cluster. Here is a minimal example policy:

...
[0] exec.0: [1521622010.013470159, {"exec"=>"ConsoleKit"}]
[1] exec.0: [1521622010.013490313, {"exec"=>"Xorg.0.log"}]
[2] exec.0: [1521622010.013492079, {"exec"=>"Xorg.0.log.old"}]
[3] exec.0: [1521622010.013493443, {"exec"=>"anaconda.ifcfg.log"}]
[4] exec.0: [1521622010.013494707, {"exec"=>"anaconda.log"}]
[5] exec.0: [1521622010.013496016, {"exec"=>"anaconda.program.log"}]
[6] exec.0: [1521622010.013497225, {"exec"=>"anaconda.storage.log"}]
...

pipeline:
  inputs:
    - name: exec
      tag: exec_ls
      command: ls /var/log
      interval_sec: 1
      interval_nsec: 0
      buf_size: 8mb
      oneshot: false

  outputs:
    - name: stdout
      match: '*'

pipeline:
  inputs:
    - name: exec
      tag: exec_oneshot_demo
      command: 'for s in $(seq 1 10); do echo "count: $s"; sleep 1; done; exit 1'
      oneshot: true
      exit_after_oneshot: true
      propagate_exit_code: true

  outputs:
    - name: stdout
      match: '*'

[INPUT]
  Name                exec
  Tag                 exec_oneshot_demo
  Command             for s in $(seq 1 10); do echo "count: $s"; sleep 1; done; exit 1
  Oneshot             true
  Exit_After_Oneshot  true
  Propagate_Exit_Code true

[OUTPUT]
  Name   stdout
  Match  *

...
[0] exec_oneshot_demo: [[1681702172.950574027, {}], {"exec"=>"count: 1"}]
[1] exec_oneshot_demo: [[1681702173.951663666, {}], {"exec"=>"count: 2"}]
[2] exec_oneshot_demo: [[1681702174.953873724, {}], {"exec"=>"count: 3"}]
[3] exec_oneshot_demo: [[1681702175.955760865, {}], {"exec"=>"count: 4"}]
[4] exec_oneshot_demo: [[1681702176.956840282, {}], {"exec"=>"count: 5"}]
[5] exec_oneshot_demo: [[1681702177.958292246, {}], {"exec"=>"count: 6"}]
[6] exec_oneshot_demo: [[1681702178.959508200, {}], {"exec"=>"count: 7"}]
[7] exec_oneshot_demo: [[1681702179.961715745, {}], {"exec"=>"count: 8"}]
[8] exec_oneshot_demo: [[1681702180.963924140, {}], {"exec"=>"count: 9"}]
[9] exec_oneshot_demo: [[1681702181.965852990, {}], {"exec"=>"count: 10"}]
...

#!/bin/bash
# This is a DANGEROUS example of what NOT to do, NEVER DO THIS
exec fluent-bit \
  -o stdout \
  -i exec \
  -p exit_after_oneshot=true \
  -p propagate_exit_code=true \
  -p command='myscript $*'

server {
  listen       80;
  listen  [::]:80;
  server_name  localhost;
  location / {
    root   /usr/share/nginx/html;
    index  index.html index.htm;
  }
  # Configure the stub status handler.
  location /status {
    stub_status;
  }
}

server {
  listen       80;
  listen  [::]:80;
  server_name  localhost;

  # Enable /api/ location with appropriate access control in order
  # to make use of NGINX Plus API.
  location /api/ {
    api write=on;
    # Configure to allow requests from the server running Fluent Bit.
    allow 192.168.1.*;
    deny all;
  }
}

pipeline:
  inputs:
    - name: nginx_metrics
      host: 127.0.0.1
      port: 80
      status_url: /status
      nginx_plus: off
      scrape_interval: 5s

  outputs:
    - name: stdout
      match: '*'

pipeline:
  inputs:
    - name: nginx_metrics
      host: 127.0.0.1
      port: 80
      status_url: /api
      nginx_plus: on
      scrape_interval: 5s

  outputs:
    - name: stdout
      match: '*'

...
2021-10-14T19:37:37.228691854Z nginx_connections_accepted = 788253884
2021-10-14T19:37:37.228691854Z nginx_connections_handled = 788253884
2021-10-14T19:37:37.228691854Z nginx_http_requests_total = 42045501
2021-10-14T19:37:37.228691854Z nginx_connections_active = 2009
2021-10-14T19:37:37.228691854Z nginx_connections_reading = 0
2021-10-14T19:37:37.228691854Z nginx_connections_writing = 1
2021-10-14T19:37:37.228691854Z nginx_connections_waiting = 2008
2021-10-14T19:37:35.229919621Z nginx_up = 1
...

parsers:
    - name: docker
      format: json
      time_key: time
      time_format: '%Y-%m-%dT%H:%M:%S.%L'
      time_keep: true

pipeline:
    inputs:
        - name: tail
          tag: kube.*
          path: 'C:\\var\\log\\containers\\*.log'
          parser: docker
          db: 'C:\\fluent-bit\\tail_docker.db'
          mem_buf_limit: 7MB
          refresh_interval: 10

        - name: tail
          tag: kube.error
          path: 'C:\\k\\kubelet.err.log'
          db: 'C:\\fluent-bit\\tail_kubelet.db'

    filters:
        - name: kubernetes
          match: kube.*
          kube_url: 'https://kubernetes.default.svc.cluster.local:443'

    outputs:
        - name: stdout
          match: '*'

fluent-bit.conf: |
    [SERVICE]
      Parsers_File      C:\\fluent-bit\\parsers.conf

    [INPUT]
      Name              tail
      Tag               kube.*
      Path              C:\\var\\log\\containers\\*.log
      Parser            docker
      DB                C:\\fluent-bit\\tail_docker.db
      Mem_Buf_Limit     7MB
      Refresh_Interval  10

    [INPUT]
      Name              tail
      Tag               kubelet.err
      Path              C:\\k\\kubelet.err.log
      DB                C:\\fluent-bit\\tail_kubelet.db

    [FILTER]
      Name              kubernetes
      Match             kube.*
      Kube_URL          https://kubernetes.default.svc.cluster.local:443

    [OUTPUT]
      Name  stdout
      Match *

parsers.conf: |
    [PARSER]
        Name         docker
        Format       json
        Time_Key     time
        Time_Format  %Y-%m-%dT%H:%M:%S.%L
        Time_Keep    On

spec:
  containers:
  - name: fluent-bit
    image: my-repo/fluent-bit:1.8.4
    volumeMounts:
    - mountPath: C:\k
      name: k
    - mountPath: C:\var\log
      name: varlog
    - mountPath: C:\ProgramData
      name: progdata
  volumes:
  - name: k
    hostPath:
      path: C:\k
  - name: varlog
    hostPath:
      path: C:\var\log
  - name: progdata
    hostPath:
      path: C:\ProgramData

service:
  flush: 1
  log_level: info
  http_server: true
  http_listen: 0.0.0.0
  http_port: 2020
  hot_reload: on

pipeline:
  inputs:
    - name: random

  outputs:
    - name: stdout
      match: '*'

pipeline:
  inputs:
    - name: forward
      listen: 0.0.0.0
      port: 24224
      buffer_chunk_size: 1M
      buffer_max_size: 6M
      security.users: fluentbit changeme
      shared_key: secret
      self_hostname: flb.server.local

  outputs:
    - name: stdout
      match: '*'

[INPUT]
  Name              forward
  Listen            0.0.0.0
  Port              24224
  Buffer_Chunk_Size 1M
  Buffer_Max_Size   6M
  Security.Users    fluentbit changeme
  Shared_Key        secret
  Self_Hostname     flb.server.local

[OUTPUT]
  Name   stdout
  Match  *

pipeline:
  inputs:
    - name: forward
      listen: 0.0.0.0
      port: 24224
      buffer_chunk_size: 1M
      buffer_max_size: 6M
      security.users: fluentbit changeme
      empty_shared_key: true
      self_hostname: flb.server.local

  outputs:
    - name: stdout
      match: '*'

[INPUT]
  Name              forward
  Listen            0.0.0.0
  Port              24224
  Buffer_Chunk_Size 1M
  Buffer_Max_Size   6M
  Security.Users    fluentbit changeme
  Empty_Shared_Key  true
  Self_Hostname     flb.server.local

[OUTPUT]
  Name   stdout
  Match  *

pipeline:
  inputs:
    - name: gpu_metrics
      cards_exclude: "0"
      cards_include: "1"
      enable_power: true
      enable_temperature: true
      path_sysfs: /sys
      scrape_interval: 2

  outputs:
    - name: stdout
      match: '*'

[INPUT]
  Name                gpu_metrics
  Cards_Exclude       0
  Cards_Include       1
  Enable_Power        true
  Enable_Temperature  true
  Path_Sysfs          /sys
  Scrape_Interval     2

[OUTPUT]
  Name   stdout
  Match  *

03:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Navi 31 [Radeon RX 7900 XT/7900 XTX/7900 GRE/7900M] (rev ce)
73:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Granite Ridge [Radeon Graphics] (rev c5)

2025-10-25T20:36:55.236905093Z gpu_utilization_percent{card="1",vendor="amd"} = 2
2025-10-25T20:36:55.237853918Z gpu_utilization_percent{card="0",vendor="amd"} = 0
2025-10-25T20:36:55.236905093Z gpu_memory_used_bytes{card="1",vendor="amd"} = 1580118016
2025-10-25T20:36:55.237853918Z gpu_memory_used_bytes{card="0",vendor="amd"} = 26083328
2025-10-25T20:36:55.236905093Z gpu_memory_total_bytes{card="1",vendor="amd"} = 17163091968
2025-10-25T20:36:55.237853918Z gpu_memory_total_bytes{card="0",vendor="amd"} = 2147483648
2025-10-25T20:36:55.236905093Z gpu_clock_mhz{card="1",vendor="amd",type="graphics"} = 45
2025-10-25T20:36:55.236905093Z gpu_clock_mhz{card="1",vendor="amd",type="memory"} = 96
2025-10-25T20:36:55.236905093Z gpu_clock_mhz{card="1",vendor="amd",type="soc"} = 500
2025-10-25T20:36:55.237853918Z gpu_clock_mhz{card="0",vendor="amd",type="graphics"} = 600
2025-10-25T20:36:55.237853918Z gpu_clock_mhz{card="0",vendor="amd",type="memory"} = 2800
2025-10-25T20:36:55.237853918Z gpu_clock_mhz{card="0",vendor="amd",type="soc"} = 1200
2025-10-25T20:36:55.236905093Z gpu_power_watts{card="1",vendor="amd"} = 28
2025-10-25T20:36:55.236905093Z gpu_temperature_celsius{card="1",vendor="amd"} = 28
2025-10-25T20:36:55.237853918Z gpu_temperature_celsius{card="0",vendor="amd"} = 39
2025-10-25T20:36:55.236905093Z gpu_fan_speed_rpm{card="1",vendor="amd"} = 0
2025-10-25T20:36:55.236905093Z gpu_fan_pwm_percent{card="1",vendor="amd"} = 0

pipeline:
  inputs:
    - name: prometheus_textfile
      tag: custom_metrics
      path: '/var/lib/prometheus/textfile/*.prom'
      scrape_interval: 15
  outputs:
    - name: prometheus_exporter
      match: custom_metrics
      host: 192.168.100.61
      port: 2021

# HELP custom_counter_total A custom counter metric
# TYPE custom_counter_total counter
custom_counter_total{instance="server1",job="myapp"} 42

# HELP custom_gauge A custom gauge metric
# TYPE custom_gauge gauge
custom_gauge{environment="production"} 1.23

# HELP custom_histogram_bucket A custom histogram
# TYPE custom_histogram_bucket histogram
custom_histogram_bucket{le="0.1"} 10
custom_histogram_bucket{le="0.5"} 25
custom_histogram_bucket{le="1.0"} 40
custom_histogram_bucket{le="+Inf"} 50
custom_histogram_sum 125.5
custom_histogram_count 50

# Script writes metrics to file
echo "# HELP app_requests_total Total HTTP requests" > /var/lib/prometheus/textfile/app.prom
echo "# TYPE app_requests_total counter" >> /var/lib/prometheus/textfile/app.prom
echo "app_requests_total{status=\"200\"} 1500" >> /var/lib/prometheus/textfile/app.prom
echo "app_requests_total{status=\"404\"} 23" >> /var/lib/prometheus/textfile/app.prom

#!/bin/bash
# Backup script writes completion metrics
BACKUP_START=$(date +%s)
# ... perform backup ...
BACKUP_END=$(date +%s)
DURATION=$((BACKUP_END - BACKUP_START))

cat > /var/lib/prometheus/textfile/backup.prom << EOF
# HELP backup_duration_seconds Time taken to complete backup
# TYPE backup_duration_seconds gauge
backup_duration_seconds ${DURATION}

# HELP backup_last_success_timestamp_seconds Last successful backup timestamp
# TYPE backup_last_success_timestamp_seconds gauge
backup_last_success_timestamp_seconds ${BACKUP_END}
EOF

pipeline:
  inputs:
    - name: prometheus_textfile
      tag: textfile_metrics
      path: /var/lib/prometheus/textfile
    - name: node_exporter_metrics
      tag: system_metrics
      scrape_interval: 15
  outputs:
    - name: opentelemetry
      match: '*'
      host: 192.168.56.4
      port: 2021

Environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)

pipeline:
  inputs:
    - name: kafka
      brokers: kafka-broker:9092
      topics: fb-source
      poll_ms: 100
      format: json

  filters:
    - name: lua
      match: '*'
      script: kafka.lua
      call: modify_kafka_message

  outputs:
    - name: kafka
      brokers: kafka-broker:9092
      topics: fb-sink

[INPUT]
  Name     kafka
  Brokers  kafka-broker:9092
  Topics   fb-source
  Poll_ms  100
  Format   json

[FILTER]
  Name    lua
  Match   *
  Script  kafka.lua
  Call    modify_kafka_message

[OUTPUT]
  Name    kafka
  Brokers kafka-broker:9092
  Topics  fb-sink

pipeline:
  inputs:
    - name: kafka
      brokers: my-cluster.abcdef.c1.kafka.us-east-1.amazonaws.com:9098
      topics: my-topic
      aws_msk_iam: true
      aws_msk_iam_cluster_arn: arn:aws:kafka:us-east-1:123456789012:cluster/my-cluster/abcdef-1234-5678-9012-abcdefghijkl-s3

  outputs:
    - name: stdout
      match: '*'

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "VisualEditor0",
      "Effect": "Allow",
      "Action": [
        "kafka-cluster:*",
        "kafka-cluster:DescribeCluster",
        "kafka-cluster:ReadData",
        "kafka-cluster:DescribeTopic",
        "kafka-cluster:Connect"
      ],
      "Resource": "*"
    }
  ]
}

[0] stdin.0: [[1684196745.942883835, {}], {"key"=>"some value"}]
[0] stdin.0: [[1684196746.938949056, {}], {"key"=>"some value"}]
[0] stdin.0: [[1684196747.940162493, {}], {"key"=>"some value"}]
[0] stdin.0: [[1684196748.941392297, {}], {"key"=>"some value"}]
[0] stdin.0: [[1684196749.942644238, {}], {"key"=>"some value"}]
[0] stdin.0: [[1684196750.943721442, {}], {"key"=>"some value"}]

[0] stdin.0: [[1684110480.028171300, {}], {"realtimestamp"=>1684196880.030070}]
[0] stdin.0: [[1684110481.033753395, {}], {"realtimestamp"=>1684196881.034741}]
[0] stdin.0: [[1684110482.036730051, {}], {"realtimestamp"=>1684196882.037704}]
[0] stdin.0: [[1684110483.039903879, {}], {"realtimestamp"=>1684196883.041081}]
[0] stdin.0: [[1684110484.044719457, {}], {"realtimestamp"=>1684196884.046404}]
[0] stdin.0: [[1684110485.048710107, {}], {"realtimestamp"=>1684196885.049651}]

#!/bin/sh
for ((i=0; i<=5; i++)); do
  echo -n "
    [
      [
        $(date '+%s.%N' -d '1 day ago'),
    {\"metakey\": \"metavalue\"}
      ],
      {
        \"realtimestamp\": $(date '+%s.%N')
      }
    ]
  "
  sleep 1
done

[0] stdin.0: [[1684110513.060139417, {"metakey"=>"metavalue"}], {"realtimestamp"=>1684196913.061017}]
[0] stdin.0: [[1684110514.063085317, {"metakey"=>"metavalue"}], {"realtimestamp"=>1684196914.064145}]
[0] stdin.0: [[1684110515.066210508, {"metakey"=>"metavalue"}], {"realtimestamp"=>1684196915.067155}]
[0] stdin.0: [[1684110516.069149971, {"metakey"=>"metavalue"}], {"realtimestamp"=>1684196916.070132}]
[0] stdin.0: [[1684110517.072484016, {"metakey"=>"metavalue"}], {"realtimestamp"=>1684196917.073636}]
[0] stdin.0: [[1684110518.075428724, {"metakey"=>"metavalue"}], {"realtimestamp"=>1684196918.076292}]

...
[0] stdin: [[1751545974.960182000, {}], {"message"=>"1"}]
[1] stdin: [[1751545974.960246000, {}], {"message"=>"2"}]
[2] stdin: [[1751545974.960255000, {}], {"message"=>"3"}]
[3] stdin: [[1751545974.960262000, {}], {"message"=>"4"}]
[4] stdin: [[1751545974.960268000, {}], {"message"=>"5"}]
...

service:
  flush: 5
  daemon: off
  log_level: info
  parsers_file: parsers.yaml
  plugins_file: plugins.yaml
  http_server

[SERVICE]
  # Flush
  # =====
  # set an interval of seconds before to flush records to a destination
  flush        5

  # Daemon
  # ======
  # instruct Fluent Bit to run in foreground or background mode.
  daemon       Off

  # Log_Level
  # =========
  # Set the verbosity level of the service, values can be:
  #
  # - error
  # - warning
  # - info
  # - debug
  # - trace

fluent-bit
├── bin
│   ├── fluent-bit.dll
│   └── fluent-bit.exe
│   └── fluent-bit.pdb
├── conf
│   ├── fluent-bit.conf
│   ├── parsers.conf
│   └── plugins.conf
└── include
    │   ├── flb_api.h
    │   ├── ...
    │   └── flb_worker.h
    └── fluent-bit.h

wget -o winflexbison.zip https://github.com/lexxmark/winflexbison/releases/download/v2.5.22/win_flex_bison-2.5.22.zip
Expand-Archive winflexbison.zip -Destination C:\WinFlexBison
cp -Path C:\WinFlexBison\win_bison.exe C:\WinFlexBison\bison.exe
cp -Path C:\WinFlexBison\win_flex.exe C:\WinFlexBison\flex.exe

fluent-bit.exe  -i dummy -o stdout

...
[0] dummy.0: [1561684385.443823800, {"message"=>"dummy"}]
[1] dummy.0: [1561684386.428399000, {"message"=>"dummy"}]
[2] dummy.0: [1561684387.443641900, {"message"=>"dummy"}]
[3] dummy.0: [1561684388.441405800, {"message"=>"dummy"}]

C:\fluent-bit\
├── conf
│   ├── fluent-bit.conf
│   └── parsers.conf
│   └── plugins.conf
└── bin
    ├── fluent-bit.dll
    └── fluent-bit.exe
    └── fluent-bit.pdb

New-Service fluent-bit -BinaryPathName "`"C:\Program Files\fluent-bit\bin\fluent-bit.exe`" -c `"C:\Program Files\fluent-bit\conf\fluent-bit.conf`"" -StartupType Automatic -Description "This service runs Fluent Bit, a log collector that enables real-time processing and delivery of log data to centralized logging systems."

get-Service fluent-bit | format-list
Name                : fluent-bit
DisplayName         : fluent-bit
Status              : Running
DependentServices   : {}
ServicesDependedOn  : {}
CanPauseAndContinue : False
CanShutdown         : False
CanStop             : True
ServiceType         : Win32OwnProcess

pipeline:
  inputs:
    - name: blob
      path: /var/log/binaries/*.bin
      database_file: /var/lib/fluent-bit/blob.db
      scan_refresh_interval: 10s
      tag: blob.files

  outputs:
    - name: stdout
      match: '*'

[INPUT]
  Name                blob
  Path                /var/log/binaries/*.bin
  Database_File       /var/lib/fluent-bit/blob.db
  Scan_Refresh_Interval 10s
  Tag                 blob.files

[OUTPUT]
  Name   stdout
  Match  *

pipeline:
  inputs:
    - name: blob
      path: /data/artifacts/**/*
      exclude_pattern: *.tmp,*.bak,*.old
      storage.type: filesystem
      storage.pause_on_chunks_overlimit: true
      mem_buf_limit: 50M
      tag: artifacts

  outputs:
    - name: stdout
      match: '*'

[INPUT]
  Name                        blob
  Path                        /data/artifacts/**/*
  Exclude_Pattern            *.tmp,*.bak,*.old
  Storage.Type                filesystem
  Storage.Pause_On_Chunks_Overlimit true
  Mem_Buf_Limit              50M
  Tag                         artifacts

[OUTPUT]
  Name   stdout
  Match  *

pipeline:
  inputs:
    - name: blob
      path: /var/log/binaries/*.bin
      database_file: /var/lib/fluent-bit/blob.db
      upload_success_action: add_suffix
      upload_success_suffix: .processed
      upload_failure_action: add_suffix
      upload_failure_suffix: .failed
      tag: blob.data

  outputs:
    - name: stdout
      match: '*'

[INPUT]
  Name                  blob
  Path                  /var/log/binaries/*.bin
  Database_File         /var/lib/fluent-bit/blob.db
  Upload_Success_Action add_suffix
  Upload_Success_Suffix .processed
  Upload_Failure_Action add_suffix
  Upload_Failure_Suffix .failed
  Tag                   blob.data

[OUTPUT]
  Name   stdout
  Match  *

Buffering and storage

Fluent Bit collects, parses, filters, and ships logs to a central place. A critical piece of this workflow is the ability to do buffering: a mechanism to place processed data into a temporary location until is ready to be shipped.

By default when Fluent Bit processes data, it uses Memory as a primary and temporary place to store the records. There are scenarios where it would be ideal to have a persistent buffering mechanism based in the filesystem to provide aggregation and data safety capabilities.

Choosing the right configuration is critical and the behavior of the service can be conditioned based in the backpressure settings. Before jumping into the configuration it helps to understand the relationship between chunks, memory, filesystem, and backpressure.

Chunks, memory, filesystem, and backpressure

Understanding chunks, buffering, and backpressure is critical for a proper configuration.

Backpressure

See for a full explanation.

Chunks

When an input plugin source emits records, the engine groups the records together in a chunk. A chunk's size usually is around 2 MB. By configuration, the engine decides where to place this chunk. By default, all chunks are created only in memory.

Irrecoverable chunks

There are two scenarios where Fluent Bit marks chunks as irrecoverable:

When Fluent Bit encounters a bad layout in a chunk. A bad layout is a chunk that doesn't conform to the expected format.
When Fluent Bit encounters an incorrect or invalid chunk header size.

In both scenarios Fluent Bit logs an error message and then discards the irrecoverable chunks.

Buffering and memory

As mentioned previously, chunks generated by the engine are placed in memory by default, but this is configurable.

If memory is the only mechanism set for the input plugin, it will store as much data as possible in memory. This is the fastest mechanism with the least system overhead. However, if the service isn't able to deliver the records fast enough, Fluent Bit memory usage increases as it accumulates more data than it can deliver.

In a high load environment with backpressure, having high memory usage risks getting killed by the kernel's OOM Killer. To work around this backpressure scenario, limit the amount of memory in records that an input plugin can register using the mem_buf_limit property. If a plugin has queued more than the mem_buf_limit, it won't be able to ingest more until that data can be delivered or flushed properly. In this scenario the input plugin in question is paused. When the input is paused, records won't be ingested until the plugin resumes. For some inputs, such as TCP and tail, pausing the input will almost certainly lead to log loss. For the tail input, Fluent Bit can save its current offset in the current file it's reading, and pick back up when the input resumes.

Look for messages in the Fluent Bit log output like:

Using mem_buf_limit is good for certain scenarios and environments. It helps to control the memory usage of the service. However, if a file rotates while the plugin is paused, data can be lost since it won't be able to register new records. This can happen with any input source plugin. The goal of mem_buf_limit is memory control and survival of the service.

For a full data safety guarantee, use filesystem buffering.

Choose your preferred format for an example input definition:

If this input uses more than 50 MB memory to buffer logs, you will get a warning like this in the Fluent Bit logs:

mem_buf_Limit applies only when storage.type is set to the default value of memory.

Filesystem buffering

Filesystem buffering helps with backpressure and overall memory control. Enable it using storage.type filesystem.

Memory and filesystem buffering mechanisms aren't mutually exclusive. Enabling filesystem buffering for your input plugin source can improve both performance and data safety.

Enabling filesystem buffering changes the behavior of the engine. Upon chunk creation, the engine stores the content in memory and also maps a copy on disk through . The newly created chunk is active in memory, backed up on disk, and called to be up, which means the chunk content is up in memory.

Fluent Bit controls the number of chunks that are up in memory by using the filesystem buffering mechanism to deal with high memory usage and backpressure.

By default, the engine allows a total of 128 chunks up in memory in total, considering all chunks. This value is controlled by the service property storage.max_chunks_up. The active chunks that are up are either ready for delivery (marked busy and locked), or are still receiving records. Any other remaining chunk is in a down state, which means that it's only in the filesystem and won't be up in memory unless it's ready to be delivered. Chunks are never much larger than 2 MB, so with the default storage.max_chunks_up value of 128, each input is limited to roughly 256 MB of memory.

If the input plugin has enabled storage.type as filesystem, when reaching the storage.max_chunks_up threshold, instead of the plugin being paused, all new data will go to chunks that are down in the filesystem. This lets you control memory usage by the service and also provides a guarantee that the service won't lose any data. By default, the enforcement of the storage.max_chunks_up limit is best-effort. Fluent Bit can only append new data to chunks that are up. When the limit is reached chunks will be temporarily brought up in memory to ingest new data, and then put to a down state afterwards. In general, Fluent Bit works to keep the total number of up chunks at or under storage.max_chunks_up.

If storage.pause_on_chunks_overlimit is enabled (default is off), the input plugin pauses upon exceeding storage.max_chunks_up. With this option, storage.max_chunks_up becomes a hard limit for the input. When the input is paused, records won't be ingested until the plugin resumes. For some inputs, such as TCP and tail, pausing the input will almost certainly lead to log loss. For the tail input, Fluent Bit can save its current offset in the current file it's reading, and pick back up when the input is resumed.

Look for messages in the Fluent Bit log output like:

Limiting filesystem space for chunks

Fluent Bit implements the concept of logical queues. Based on its tag, a chunk can be routed to multiple destinations. Fluent Bit keeps an internal reference from where a chunk was created and where it needs to go.

It's common to find cases where multiple destinations with different response times exist for a chunk, or one of the destinations is generating backpressure.

To limit the amount of filesystem chunks logically queueing, Fluent Bit v1.6 and later includes the storage.total_limit_size configuration property for output. This property limits the total size in bytes of chunks that can exist in the filesystem for a certain logical output destination. If one of the destinations reaches the configured storage.total_limit_size, the oldest chunk from its queue for that logical output destination will be discarded to make room for new data.

Configuration

The storage layer configuration takes place in three sections:

Service
Input
Output

The known Service section configures a global environment for the storage layer, the Input sections define which buffering mechanism to use, and the Output defines limits for the logical filesystem queues.

Service section configuration

The Service section refers to the section defined in the main :

Key

Description

Default

Dead letter queue (DLQ)

The Dead Letter Queue (DLQ) feature preserves chunks that fail to be delivered to output destinations. Instead of losing this data, Fluent Bit copies the rejected chunks to a dedicated storage location for later analysis and troubleshooting.

When dead letter queue is triggered

Chunks are copied to the DLQ in the following failure scenarios:

Permanent errors: When an output plugin returns an unrecoverable error (FLB_ERROR).
Retry limit reached: When a chunk exhausts all configured retry attempts.
Retries disabled: When retry_limit is set to no_retries and a flush fails.

Requirements

The DLQ feature requires:

storage.path must be configured (filesystem storage must be enabled).
storage.keep.rejected must be set to On.

Dead letter queue file location and format

Rejected chunks are stored in a subdirectory under storage.path. For example, with the following configuration:

Rejected chunks are stored at /var/log/flb-storage/rejected/.

Each DLQ file is named using this format:

For example: kube_var_log_containers_test_400_http_0x7f8b4c.flb

The file contains the original chunk data in the internal format of Fluent Bit, preserving all records and metadata.

Troubleshooting with dead letter queue

The DLQ feature enables the following capabilities:

Data preservation: Invalid or rejected chunks are preserved instead of being permanently lost.
Root cause analysis: Investigate why specific data failed to be delivered without impacting live processing.
Data recovery: Replay or transform rejected chunks after fixing the underlying issue.
Debugging: Analyze the exact content of problematic records.

To examine DLQ chunks, you can use the storage metrics endpoint (when storage.metrics is enabled) or directly inspect the files in the rejected directory.

DLQ files remain on disk until manually removed. Monitor disk usage in the rejected directory and implement a cleanup policy for older files.

A Service section will look like this:

This configuration sets an optional buffering mechanism where the route to the data is /var/log/flb-storage/. It uses normal synchronization mode, without running a checksum and up to a maximum of 5 MB of memory when processing backlog data. Additionally, the dead letter queue is enabled, and rejected chunks are stored in /var/log/flb-storage/rejected/.

Input section configuration

Optionally, any Input plugin can configure their storage preference. The following table describes the options available:

Key

Description

Default

The following example configures a service offering filesystem buffering capabilities and two input plugins being the first based in filesystem and the second with memory only.

Output section configuration

If certain chunks are filesystem storage.type based, it's possible to control the size of the logical queue for an output plugin. The following table describes the options available:

Key

Description

Default

The following example creates records with CPU usage samples in the filesystem which are delivered to Google Stackdriver service while limiting the logical queue (buffering) to 5M:

If Fluent Bit is offline because of a network issue, it will continue buffering CPU samples, keeping a maximum of 5 MB of the newest data.

TLS

Fluent Bit provides integrated support for Transport Layer Security (TLS) and its predecessor Secure Sockets Layer (SSL). This section refers only to TLS for both implementations.

Fluent Bit 4.1.0 introduced the replacement of Next Protocol Negotiation (NPN) with Application Layer Protocol Negotiation (ALPN) as its implementation for TLS. Both NPN and ALPN are used when client and server are establishing SSL/TLS connections. ALPN avoids an additional round trip because the client list the application layer protocols supported by the client in the client hello message.

Both input and output plugins that perform Network I/O can optionally enable TLS and configure the behavior. The following table describes the properties available:

Property

Description

Default

To use TLS on input plugins, you must provide both a certificate and a private key.

The listed properties can be enabled in the configuration file, specifically in each output plugin section or directly through the command line.

The following output plugins can take advantage of the TLS feature:

The following input plugins can take advantage of the TLS feature:

In addition, other plugins implement a subset of TLS support, with restricted configuration:

Example: enable TLS on HTTP input

By default, the HTTP input plugin uses plain TCP. Run the following command to enable TLS:

See the Tips and Tricks section for details on generating self_signed.crt and self_signed.key files shown in these examples.

In the previous command, the two properties tls and tls.verify are set for demonstration purposes. Always enable verification in production environments.

The same behavior can be accomplished using a configuration file:

Example: enable TLS on HTTP output

By default, the HTTP output plugin uses plain TCP. Run the following command to enable TLS:

In the previous command, the properties tls and tls.verify are enabled for demonstration purposes. Always enable verification in production environments.

The same behavior can be accomplished using a configuration file:

Tips and tricks

Generate a self-signed certificates for testing purposes

The following command generates a 4096-bit RSA key pair and a certificate that's signed using SHA-256 with the expiration date set to 30 days in the future. In this example, test.host.net is set as the common name. This example opts out of DES, so the private key is stored in plain text.

Connect to virtual servers using TLS

Fluent Bit supports . If you are serving multiple host names on a single IP address (for example, using virtual hosting), you can make use of tls.vhost to connect to a specific hostname.

Verify `subjectAltName`

By default, TLS verification of host names isn't done automatically. As an example, you can extract the X509v3 Subject Alternative Name from a certificate:

This certificate covers only my.fluent-aggregator.net so if you use a different hostname it should fail.

To fully verify the alternative name and demonstrate the failure, enable tls.verify_hostname:

This outgoing connect will fail and disconnect:

fluent-bit -i http \
           -p port=9999 \
           -p tls=on \
           -p tls.verify=off \
           -p tls.crt_file=self_signed.crt \
           -p tls.key_file=self_signed.key \
           -o stdout \
           -m '*'

pipeline:
  inputs:
    - name: http
      port: 9999
      tls: on
      tls.verify: off
      tls.crt_file: self_signed.crt
      tls.key_file: self_signed.key

  outputs:
    - name: stdout
      match: '*'

openssl req -x509 \
            -newkey rsa:4096 \
            -sha256 \
            -nodes \
            -keyout self_signed.key \
            -out self_signed.crt \
            -subj "/CN=test.host.net"

pipeline:
  inputs:
    - name: cpu
      tag: cpu

  outputs:
    - name: forward
      match: '*'
      host: 192.168.10.100
      port: 24224
      tls: on
      tls.verify: off
      tls.ca_file: '/etc/certs/fluent.crt'
      tls.vhost: 'fluent.example.com'

[INPUT]
  Name  cpu
  Tag   cpu

[OUTPUT]
  Name        forward
  Match       *
  Host        192.168.10.100
  Port        24224
  tls         on
  tls.verify  on
  tls.ca_file /etc/certs/fluent.crt
  tls.vhost   fluent.example.com

pipeline:
  inputs:
    - name: cpu
      tag: cpu

  outputs:
    - name: forward
      match: '*'
      host: other.fluent-aggregator.net
      port: 24224
      tls: on
      tls.verify: on
      tls.verify_hostname: on
      tls.ca_file: '/path/to/fluent-x509v3-alt-name.crt'

[INPUT]
  Name  cpu
  Tag   cpu

[OUTPUT]
  Name                forward
  Match               *
  Host                other.fluent-aggregator.net
  Port                24224
  tls                 on
  tls.verify          on
  tls.verify_hostname on
  tls.ca_file         /path/to/fluent-x509v3-alt-name.crt

[2024/06/17 16:51:31] [error] [tls] error: unexpected EOF with reason: certificate verify failed
[2024/06/17 16:51:31] [debug] [upstream] connection #50 failed to other.fluent-aggregator.net:24224
[2024/06/17 16:51:31] [error] [output:forward:forward.0] no upstream connections available

service:
  flush: 1
  log_level: info
  storage.path: /var/log/flb-storage/
  storage.sync: normal
  storage.checksum: off
  storage.backlog.mem_limit: 5M
  storage.backlog.flush_on_shutdown: off
  storage.keep.rejected: on
  storage.rejected.path: rejected

[SERVICE]
  flush                     1
  log_Level                 info
  storage.path              /var/log/flb-storage/
  storage.sync              normal
  storage.checksum          off
  storage.backlog.mem_limit 5M
  storage.backlog.flush_on_shutdown off
  storage.keep.rejected     on
  storage.rejected.path     rejected

service:
  flush: 1
  log_level: info
  storage.path: /var/log/flb-storage/
  storage.sync: normal
  storage.checksum: off
  storage.max_chunks_up: 128
  storage.backlog.mem_limit: 5M

pipeline:
  inputs:
    - name: cpu
      storage.type: filesystem

    - name: mem
      storage.type: memory

[SERVICE]
  flush                     1
  log_Level                 info
  storage.path              /var/log/flb-storage/
  storage.sync              normal
  storage.checksum          off
  storage.max_chunks_up     128
  storage.backlog.mem_limit 5M

[INPUT]
  name          cpu
  storage.type  filesystem

[INPUT]
  name          mem
  storage.type  memory

service:
  flush: 1
  log_level: info
  storage.path: /var/log/flb-storage/
  storage.sync: normal
  storage.checksum: off
  storage.max_chunks_up: 128
  storage.backlog.mem_limit: 5M

pipeline:
  inputs:
    - name: cpu
      storage.type: filesystem

  outputs:
    - name: stackdriver
      match: '*'
      storage.total_limit_size: 5M

[SERVICE]
  flush                     1
  log_Level                 info
  storage.path              /var/log/flb-storage/
  storage.sync              normal
  storage.checksum          off
  storage.max_chunks_up     128
  storage.backlog.mem_limit 5M

[INPUT]
  name                      cpu
  storage.type              filesystem

[OUTPUT]
  name                      stackdriver
  match                     *
  storage.total_limit_size  5M

service:
  storage.path: /var/log/flb-storage/
  storage.keep.rejected: on
  storage.rejected.path: rejected

[SERVICE]
  storage.path          /var/log/flb-storage/
  storage.keep.rejected on
  storage.rejected.path rejected

OpenTelemetry

An input plugin to ingest OpenTelemetry logs, metrics, and traces

The OpenTelemetry input plugin lets you receive data based on the OpenTelemetry specification from various OpenTelemetry exporters, the OpenTelemetry Collector, or the Fluent Bit OpenTelemetry output plugin.

Fluent Bit has a compliant implementation which fully supports OTLP/HTTP and OTLP/GRPC. The single port configured defaults to 4318 and supports both transport methods.

Configuration

Key

Description

Default

When raw_traces is set to false (default), the traces endpoint (/v1/traces) processes incoming trace data using the unified JSON parser with strict validation. The endpoint accepts both protobuf and JSON encoded payloads. When raw_traces is set to true, any data forwarded to the traces endpoint will be packed and forwarded as a log message without processing, validation, or conversion to the Fluent Bit internal trace format.

OpenTelemetry transport protocol endpoints

Fluent Bit exposes the following endpoints for data ingestion based on the OpenTelemetry protocol:

For OTLP/HTTP:

Logs
- /v1/logs
Metrics
- /v1/metrics

For OTLP/GRPC:

Logs
- /opentelemetry.proto.collector.log.v1.LogService/Export
- /opentelemetry.proto.collector.log.v1.LogService/Export

Get started

The OpenTelemetry input plugin supports the following telemetry data types:

Type

HTTP1/JSON

HTTP1/Protobuf

HTTP2/gRPC

A sample configuration file to get started will look something like the following:

With this configuration, Fluent Bit listens on port 4318 for data. You can now send telemetry data to the endpoints /v1/metrics for metrics, /v1/traces for traces, and /v1/logs for logs.

A sample curl request to POST JSON encoded log data would be:

OpenTelemetry trace improvements

Fluent Bit includes enhanced support for OpenTelemetry traces with improved JSON parsing, error handling, and validation capabilities.

Unified trace JSON parser

Fluent Bit provides a unified interface for processing OpenTelemetry trace data in JSON format. The parser converts OpenTelemetry JSON trace payloads into the Fluent Bit internal trace representation, supporting the full OpenTelemetry trace specification including:

Resource spans with attributes
Instrumentation scope information
Span data (names, IDs, timestamps, status)
Span events and links

The unified parser handles the OpenTelemetry JSON encoding format, which wraps attribute values in type-specific containers (for example, stringValue, intValue, doubleValue, boolValue).

Error status propagation

The OpenTelemetry input plugin provides detailed error status information when processing trace data. If trace processing fails, the plugin returns specific error codes that help identify the issue:

FLB_OTEL_TRACES_ERR_INVALID_JSON - Invalid JSON format
FLB_OTEL_TRACES_ERR_INVALID_TRACE_ID - Invalid trace ID format or length
FLB_OTEL_TRACES_ERR_INVALID_SPAN_ID - Invalid span ID format or length

Valid span status codes

The OpenTelemetry specification defines three valid span status codes. When processing trace data, the plugin accepts the following status code values (case-insensitive):

OK - The operation completed successfully
ERROR - The operation has an error
UNSET - The status isn't set (default)

Any other status code value triggers FLB_OTEL_TRACES_ERR_STATUS_FAILURE and causes the trace data to be rejected. The status code must be provided as a string in the status.code field of the span object.

Error handling behavior

When trace validation fails, the following behavior applies:

Trace data is dropped: Invalid trace data isn't processed or forwarded. The trace payload is rejected immediately.
Error logging: The plugin logs an error message with the specific error status code to help diagnose issues. Error messages include the error code number and description.
No retry mechanism: Failed requests aren't automatically retried. The client must resend corrected trace data.
HTTP response codes

Strict ID decoding

Fluent Bit enforces strict validation for trace and span IDs to ensure data integrity:

Trace IDs: Must be exactly 32 hexadecimal characters (16 bytes)
Span IDs: Must be exactly 16 hexadecimal characters (8 bytes)
Parent Span IDs: Must be exactly 16 hexadecimal characters (8 bytes) when present

The validation process:

Verifies the ID length matches the expected size
Validates that all characters are valid hexadecimal digits (0-9, a-f, A-F)
Decodes the hexadecimal string to binary format
Rejects invalid IDs with appropriate error codes

Invalid IDs result in error status codes (FLB_OTEL_TRACES_ERR_INVALID_TRACE_ID, FLB_OTEL_TRACES_ERR_INVALID_SPAN_ID, and so on) and the trace data is rejected to prevent processing of corrupted or malformed trace information.

Example: JSON trace payload

The following example shows a valid OpenTelemetry JSON trace payload that can be sent to the /v1/traces endpoint:

Trace IDs must be exactly 32 hex characters and span IDs must be exactly 16 hex characters. Invalid IDs will be rejected with appropriate error messages.

In the example, the status.code field uses "OK". Valid status code values are "OK", "ERROR", and "UNSET" (case-insensitive). Any other value triggers FLB_OTEL_TRACES_ERR_STATUS_FAILURE and causes the trace to be rejected.

docker run --rm -ti fluent/fluent-bit:latest --help | grep trace
  -Z, --enable-chunk-traceenable chunk tracing, it can be activated either through the http api or the command line
  --trace-input           input to start tracing on startup.
  --trace-output          output to use for tracing on startup.
  --trace-output-property set a property for output tracing on startup.
  --trace                 setup a trace pipeline on startup. Uses a single line, ie: "input=dummy.0 output=stdout output.format='json'"

$ fluent-bit -Z -i dummy -o stdout -f 1 --trace-input=dummy.0 --trace-output=stdout

...
[0] dummy.0: [[1689971222.068537501, {}], {"message"=>"dummy"}]
[0] dummy.0: [[1689971223.068556121, {}], {"message"=>"dummy"}]
[0] trace: [[1689971222.068677045, {}], {"type"=>1, "trace_id"=>"0", "plugin_instance"=>"dummy.0", "records"=>[{"timestamp"=>1689971222, "record"=>{"message"=>"dummy"}}], "start_time"=>1689971222, "end_time"=>1689971222}]
[1] trace: [[1689971222.068735577, {}], {"type"=>3, "trace_id"=>"0", "plugin_instance"=>"dummy.0", "records"=>[{"timestamp"=>1689971222, "record"=>{"message"=>"dummy"}}], "start_time"=>1689971222, "end_time"=>1689971222}]
[0] dummy.0: [[1689971224.068586317, {}], {"message"=>"dummy"}]
[0] trace: [[1689971223.068626923, {}], {"type"=>1, "trace_id"=>"1", "plugin_instance"=>"dummy.0", "records"=>[{"timestamp"=>1689971223, "record"=>{"message"=>"dummy"}}], "start_time"=>1689971223, "end_time"=>1689971223}]
[1] trace: [[1689971223.068675735, {}], {"type"=>3, "trace_id"=>"1", "plugin_instance"=>"dummy.0", "records"=>[{"timestamp"=>1689971223, "record"=>{"message"=>"dummy"}}], "start_time"=>1689971223, "end_time"=>1689971223}]
[2] trace: [[1689971224.068689341, {}], {"type"=>1, "trace_id"=>"2", "plugin_instance"=>"dummy.0", "records"=>[{"timestamp"=>1689971224, "record"=>{"message"=>"dummy"}}], "start_time"=>1689971224, "end_time"=>1689971224}]
[3] trace: [[1689971224.068747182, {}], {"type"=>3, "trace_id"=>"2", "plugin_instance"=>"dummy.0", "records"=>[{"timestamp"=>1689971224, "record"=>{"message"=>"dummy"}}], "start_time"=>1689971224, "end_time"=>1689971224}]
^C[2023/07/21 16:27:05] [engine] caught signal (SIGINT)
[2023/07/21 16:27:05] [ warn] [engine] service will shutdown in max 5 seconds
[2023/07/21 16:27:05] [ info] [input] pausing dummy.0
[0] dummy.0: [[1689971225.068568875, {}], {"message"=>"dummy"}]
[2023/07/21 16:27:06] [ info] [engine] service has stopped (0 pending tasks)
[2023/07/21 16:27:06] [ info] [input] pausing dummy.0
[2023/07/21 16:27:06] [ warn] [engine] service will shutdown in max 1 seconds
[0] trace: [[1689971225.068654038, {}], {"type"=>1, "trace_id"=>"3", "plugin_instance"=>"dummy.0", "records"=>[{"timestamp"=>1689971225, "record"=>{"message"=>"dummy"}}], "start_time"=>1689971225, "end_time"=>1689971225}]
[1] trace: [[1689971225.068695829, {}], {"type"=>3, "trace_id"=>"3", "plugin_instance"=>"dummy.0", "records"=>[{"timestamp"=>1689971225, "record"=>{"message"=>"dummy"}}], "start_time"=>1689971225, "end_time"=>1689971225}]
[2023/07/21 16:27:07] [ info] [engine] service has stopped (0 pending tasks)
[2023/07/21 16:27:07] [ info] [output:stdout:stdout.0] thread worker #0 stopping...
[2023/07/21 16:27:07] [ info] [output:stdout:stdout.0] thread worker #0 stopped
[2023/07/21 16:27:07] [ info] [output:stdout:stdout.0] thread worker #0 stopping...
[2023/07/21 16:27:07] [ info] [output:stdout:stdout.0] thread worker #0 stopped

$ fluent-bit -Z -i dummy -o stdout -f 1 --trace-input=dummy.0 --trace-output=stdout --trace-output-property=format=json_lines

...
[0] dummy.0: [[1689971340.068565891, {}], {"message"=>"dummy"}]
[0] dummy.0: [[1689971341.068632477, {}], {"message"=>"dummy"}]
{"date":1689971340.068745,"type":1,"trace_id":"0","plugin_instance":"dummy.0","records":[{"timestamp":1689971340,"record":{"message":"dummy"}}],"start_time":1689971340,"end_time":1689971340}
{"date":1689971340.068825,"type":3,"trace_id":"0","plugin_instance":"dummy.0","records":[{"timestamp":1689971340,"record":{"message":"dummy"}}],"start_time":1689971340,"end_time":1689971340}
[0] dummy.0: [[1689971342.068613646, {}], {"message"=>"dummy"}]

$ docker run --rm -ti -p 2020:2020 fluent/fluent-bit:latest -Z -H -i dummy -p alias=input_dummy -o stdout -f 1

...
[0] dummy.0: [1666346597.203307010, {"message"=>"dummy"}]
[0] dummy.0: [1666346598.204103793, {"message"=>"dummy"}]

...
[0] dummy.0: [1666346616.203551736, {"message"=>"dummy"}]
[0] trace: [1666346617.205221952, {"type"=>1, "trace_id"=>"trace.0", "plugin_instance"=>"dummy.0", "plugin_alias"=>"input_dummy", "records"=>[{"timestamp"=>1666346617, "record"=>{"message"=>"dummy"}}], "start_time"=>1666346617, "end_time"=>1666346617}]
[0] dummy.0: [1666346617.205131790, {"message"=>"dummy"}]
[0] trace: [1666346617.205419358, {"type"=>3, "trace_id"=>"trace.0", "plugin_instance"=>"dummy.0", "plugin_alias"=>"input_dummy", "records"=>[{"timestamp"=>1666346617, "record"=>{"message"=>"dummy"}}], "start_time"=>1666346617, "end_time"=>1666346617}]
[0] trace: [1666346618.204110867, {"type"=>1, "trace_id"=>"trace.1", "plugin_instance"=>"dummy.0", "plugin_alias"=>"input_dummy", "records"=>[{"timestamp"=>1666346618, "record"=>{[0] dummy.0: [1666346618.204049246, {"message"=>"dummy"}]
"message"=>"dummy"}}], "start_time"=>1666346618, "end_time"=>1666346618}]
[0] trace: [1666346618.204198654, {"type"=>3, "trace_id"=>"trace.1", "plugin_instance"=>"dummy.0", "plugin_alias"=>"input_dummy", "records"=>[{"timestamp"=>1666346618, "record"=>{"message"=>"dummy"}}], "start_time"=>1666346618, "end_time"=>1666346618}]

$ docker run --rm -ti -p 2020:2020 fluent/fluent-bit:latest \
      -Z -H \
      -i dummy -p alias=dummy_0 -p \
         dummy='{"dummy": "dummy_0", "key_name": "foo", "key_cnt": "1"}' \
      -i dummy -p alias=dummy_1 -p dummy='{"dummy": "dummy_1"}' \
      -i dummy -p alias=dummy_2 -p dummy='{"dummy": "dummy_2"}' \
      -F record_modifier -m 'dummy.0' -p record="powered_by fluent" \
      -F record_modifier -m 'dummy.1' -p record="powered_by fluent-bit" \
      -F nest -m 'dummy.0' \
      -p operation=nest -p wildcard='key_*' -p nest_under=data \
      -o null -m '*' -f 1

...
[0] trace: [1666349359.325597543, {"type"=>1, "trace_id"=>"trace.0", "plugin_instance"=>"dummy.0", "plugin_alias"=>"dummy_0", "records"=>[{"timestamp"=>1666349359, "record"=>{"dummy"=>"dummy_0", "key_name"=>"foo", "key_cnt"=>"1"}}], "start_time"=>1666349359, "end_time"=>1666349359}]
[0] trace: [1666349359.325723747, {"type"=>2, "start_time"=>1666349359, "end_time"=>1666349359, "trace_id"=>"trace.0", "plugin_instance"=>"record_modifier.0", "records"=>[{"timestamp"=>1666349359, "record"=>{"dummy"=>"dummy_0", "key_name"=>"foo", "key_cnt"=>"1", "powered_by"=>"fluent"}}]}]
[0] trace: [1666349359.325783954, {"type"=>2, "start_time"=>1666349359, "end_time"=>1666349359, "trace_id"=>"trace.0", "plugin_instance"=>"nest.2", "records"=>[{"timestamp"=>1666349359, "record"=>{"dummy"=>"dummy_0", "powered_by"=>"fluent", "data"=>{"key_name"=>"foo", "key_cnt"=>"1"}}}]}]
[0] trace: [1666349359.325913783, {"type"=>3, "trace_id"=>"trace.0", "plugin_instance"=>"dummy.0", "plugin_alias"=>"dummy_0", "records"=>[{"timestamp"=>1666349359, "record"=>{"dummy"=>"dummy_0", "powered_by"=>"fluent", "data"=>{"key_name"=>"foo", "key_cnt"=>"1"}}}], "start_time"=>1666349359, "end_time"=>1666349359}]
[0] trace: [1666349360.323826619, {"type"=>1, "trace_id"=>"trace.1", "plugin_instance"=>"dummy.0", "plugin_alias"=>"dummy_0", "records"=>[{"timestamp"=>1666349360, "record"=>{"dummy"=>"dummy_0", "key_name"=>"foo", "key_cnt"=>"1"}}], "start_time"=>1666349360, "end_time"=>1666349360}]
[0] trace: [1666349360.323859618, {"type"=>2, "start_time"=>1666349360, "end_time"=>1666349360, "trace_id"=>"trace.1", "plugin_instance"=>"record_modifier.0", "records"=>[{"timestamp"=>1666349360, "record"=>{"dummy"=>"dummy_0", "key_name"=>"foo", "key_cnt"=>"1", "powered_by"=>"fluent"}}]}]
[0] trace: [1666349360.323900784, {"type"=>2, "start_time"=>1666349360, "end_time"=>1666349360, "trace_id"=>"trace.1", "plugin_instance"=>"nest.2", "records"=>[{"timestamp"=>1666349360, "record"=>{"dummy"=>"dummy_0", "powered_by"=>"fluent", "data"=>{"key_name"=>"foo", "key_cnt"=>"1"}}}]}]
[0] trace: [1666349360.323926366, {"type"=>3, "trace_id"=>"trace.1", "plugin_instance"=>"dummy.0", "plugin_alias"=>"dummy_0", "records"=>[{"timestamp"=>1666349360, "record"=>{"dummy"=>"dummy_0", "powered_by"=>"fluent", "data"=>{"key_name"=>"foo", "key_cnt"=>"1"}}}], "start_time"=>1666349360, "end_time"=>1666349360}]
[0] trace: [1666349361.324223752, {"type"=>1, "trace_id"=>"trace.2", "plugin_instance"=>"dummy.0", "plugin_alias"=>"dummy_0", "records"=>[{"timestamp"=>1666349361, "record"=>{"dummy"=>"dummy_0", "key_name"=>"foo", "key_cnt"=>"1"}}], "start_time"=>1666349361, "end_time"=>1666349361}]
[0] trace: [1666349361.324263959, {"type"=>2, "start_time"=>1666349361, "end_time"=>1666349361, "trace_id"=>"trace.2", "plugin_instance"=>"record_modifier.0", "records"=>[{"timestamp"=>1666349361, "record"=>{"dummy"=>"dummy_0", "key_name"=>"foo", "key_cnt"=>"1", "powered_by"=>"fluent"}}]}]
[0] trace: [1666349361.324283250, {"type"=>2, "start_time"=>1666349361, "end_time"=>1666349361, "trace_id"=>"trace.2", "plugin_instance"=>"nest.2", "records"=>[{"timestamp"=>1666349361, "record"=>{"dummy"=>"dummy_0", "powered_by"=>"fluent", "data"=>{"key_name"=>"foo", "key_cnt"=>"1"}}}]}]
[0] trace: [1666349361.324294291, {"type"=>3, "trace_id"=>"trace.2", "plugin_instance"=>"dummy.0", "plugin_alias"=>"dummy_0", "records"=>[{"timestamp"=>1666349361, "record"=>{"dummy"=>"dummy_0", "powered_by"=>"fluent", "data"=>{"key_name"=>"foo", "key_cnt"=>"1"}}}], "start_time"=>1666349361, "end_time"=>1666349361}]
^C[2022/10/21 10:49:23] [engine] caught signal (SIGINT)
[2022/10/21 10:49:23] [ warn] [engine] service will shutdown in max 5 seconds
[2022/10/21 10:49:23] [ info] [input] pausing dummy_0
[2022/10/21 10:49:23] [ info] [input] pausing dummy_1
[2022/10/21 10:49:23] [ info] [input] pausing dummy_2
[2022/10/21 10:49:23] [ info] [engine] service has stopped (0 pending tasks)
[2022/10/21 10:49:23] [ info] [input] pausing dummy_0
[2022/10/21 10:49:23] [ info] [input] pausing dummy_1
[2022/10/21 10:49:23] [ info] [input] pausing dummy_2
[0] trace: [1666349362.323272011, {"type"=>1, "trace_id"=>"trace.3", "plugin_instance"=>"dummy.0", "plugin_alias"=>"dummy_0", "records"=>[{"timestamp"=>1666349362, "record"=>{"dummy"=>"dummy_0", "key_name"=>"foo", "key_cnt"=>"1"}}], "start_time"=>1666349362, "end_time"=>1666349362}]
[0] trace: [1666349362.323306843, {"type"=>2, "start_time"=>1666349362, "end_time"=>1666349362, "trace_id"=>"trace.3", "plugin_instance"=>"record_modifier.0", "records"=>[{"timestamp"=>1666349362, "record"=>{"dummy"=>"dummy_0", "key_name"=>"foo", "key_cnt"=>"1", "powered_by"=>"fluent"}}]}]
[0] trace: [1666349362.323323884, {"type"=>2, "start_time"=>1666349362, "end_time"=>1666349362, "trace_id"=>"trace.3", "plugin_instance"=>"nest.2", "records"=>[{"timestamp"=>1666349362, "record"=>{"dummy"=>"dummy_0", "powered_by"=>"fluent", "data"=>{"key_name"=>"foo", "key_cnt"=>"1"}}}]}]
[0] trace: [1666349362.323334509, {"type"=>3, "trace_id"=>"trace.3", "plugin_instance"=>"dummy.0", "plugin_alias"=>"dummy_0", "records"=>[{"timestamp"=>1666349362, "record"=>{"dummy"=>"dummy_0", "powered_by"=>"fluent", "data"=>{"key_name"=>"foo", "key_cnt"=>"1"}}}], "start_time"=>1666349362, "end_time"=>1666349362}]
[2022/10/21 10:49:24] [ warn] [engine] service will shutdown in max 1 seconds
[2022/10/21 10:49:25] [ info] [engine] service has stopped (0 pending tasks)
[2022/10/21 10:49:25] [ info] [output:stdout:stdout.0] thread worker #0 stopping...
[2022/10/21 10:49:25] [ info] [output:stdout:stdout.0] thread worker #0 stopped
[2022/10/21 10:49:25] [ info] [output:null:null.0] thread worker #0 stopping...
[2022/10/21 10:49:25] [ info] [output:null:null.0] thread worker #0 stopped

$ docker run --rm -ti -p 2020:2020 fluent/fluent-bit:latest -Z -H -i dummy -p alias=input_dummy -o stdout -f 1

...
[0] dummy.0: [1674805465.976012761, {"message"=>"dummy"}]
[0] dummy.0: [1674805466.973669512, {"message"=>"dummy"}]

...
[0] dummy.0: [1674805635.972373840, {"message"=>"dummy"}]
[{"date":1674805634.974457,"type":1,"trace_id":"0","plugin_instance":"dummy.0","plugin_alias":"input_dummy","records":[{"timestamp":1674805634,"record":{"message":"dummy"}}],"start_time":1674805634,"end_time":1674805634},{"date":1674805634.974605,"type":3,"trace_id":"0","plugin_instance":"dummy.0","plugin_alias":"input_dummy","records":[{"timestamp":1674805634,"record":{"message":"dummy"}}],"start_time":1674805634,"end_time":1674805634},{"date":1674805635.972398,"type":1,"trace_id":"1","plugin_instance":"dummy.0","plugin_alias":"input_dummy","records":[{"timestamp":1674805635,"record":{"message":"dummy"}}],"start_time":1674805635,"end_time":1674805635},{"date":1674805635.972413,"type":3,"trace_id":"1","plugin_instance":"dummy.0","plugin_alias":"input_dummy","records":[{"timestamp":1674805635,"record":{"message":"dummy"}}],"start_time":1674805635,"end_time":1674805635}]
[0] dummy.0: [1674805636.973970215, {"message"=>"dummy"}]
[{"date":1674805636.974008,"type":1,"trace_id":"2","plugin_instance":"dummy.0","plugin_alias":"input_dummy","records":[{"timestamp":1674805636,"record":{"message":"dummy"}}],"start_time":1674805636,"end_time":1674805636},{"date":1674805636.974034,"type":3,"trace_id":"2","plugin_instance":"dummy.0","plugin_alias":"input_dummy","records":[{"timestamp":1674805636,"record":{"message":"dummy"}}],"start_time":1674805636,"end_time":1674805636}]

{
  "type": 2,
  "start_time": 1666349231,
  "end_time": 1666349231,
  "trace_id": "trace.1",
  "plugin_instance": "nest.2",
  "records": [{
    "timestamp": 1666349231,
    "record": {
      "dummy": "dummy_0",
      "powered_by": "fluent",
      "data": {
        "key_name": "foo",
        "key_cnt": "1"
      }
    }
  }]
}

...
[engine] caught signal (SIGCONT)
[2020/03/23 17:39:02] Fluent Bit Dump

===== Input =====
syslog_debug (syslog)
│
├─ status
│  └─ overlimit     : no
│     ├─ mem size   : 60.8M (63752145 bytes)
│     └─ mem limit  : 61.0M (64000000 bytes)
│
├─ tasks
│  ├─ total tasks   : 92
│  ├─ new           : 0
│  ├─ running       : 92
│  └─ size          : 171.1M (179391504 bytes)
│
└─ chunks
   └─ total chunks  : 92
      ├─ up chunks  : 35
      ├─ down chunks: 57
      └─ busy chunks: 92
         ├─ size    : 60.8M (63752145 bytes)
         └─ size err: 0

===== Storage Layer =====
total chunks     : 92
├─ mem chunks    : 0
└─ fs chunks     : 92
   ├─ up         : 35
   └─ down       : 57

FLB_OTEL_TRACES_ERR_INVALID_PARENT_SPAN_ID - Invalid parent span ID

curl --header "Content-Type: application/json" --request POST --data '{"resourceLogs":[{"resource":{},"scopeLogs":[{"scope":{},"logRecords":[{"timeUnixNano":"1660296023390371588","body":{"stringValue":"{\"message\":\"dummy\"}"},"traceId":"","spanId":""}]}]}]}'   http://0.0.0.0:4318/v1/logs

{
  "resourceSpans": [
    {
      "resource": {
        "attributes": [
          {
            "key": "service.name",
            "value": {
              "stringValue": "my-service"
            }
          }
        ]
      },
      "scopeSpans": [
        {
          "scope": {
            "name": "my-instrumentation",
            "version": "1.0.0"
          },
          "spans": [
            {
              "traceId": "0123456789abcdef0123456789abcdef",
              "spanId": "0123456789abcdef",
              "name": "my-span",
              "kind": 1,
              "startTimeUnixNano": "1660296023390371588",
              "endTimeUnixNano": "1660296023391371588",
              "status": {
                "code": "OK"
              },
              "attributes": [
                {
                  "key": "http.method",
                  "value": {
                    "stringValue": "GET"
                  }
                }
              ]
            }
          ]
        }
      ]
    }
  ]
}

# Node Exporter Metrics + Prometheus Exporter
# -------------------------------------------
# The following example collect host metrics on Linux and expose
# them through a Prometheus HTTP endpoint.
#
# After starting the service try it with:
#
# $ curl http://127.0.0.1:2021/metrics
#
service:
  flush: 1
  log_level: info

pipeline:
  inputs:
    - name: node_exporter_metrics
      tag:  node_metrics
      scrape_interval: 2

  outputs:
    - name: prometheus_exporter
      match: node_metrics
      host: 0.0.0.0
      port: 2021


# Node Exporter Metrics + Prometheus Exporter
# -------------------------------------------
# The following example collect host metrics on Linux and expose
# them through a Prometheus HTTP end-point.
#
# After starting the service try it with:
#
# $ curl http://127.0.0.1:2021/metrics
#
[SERVICE]
  flush           1
  log_level       info

[INPUT]
  name            node_exporter_metrics
  tag             node_metrics
  scrape_interval 2

[OUTPUT]
  name            prometheus_exporter
  match           node_metrics
  host            0.0.0.0
  port            2021

docker run -ti -v /proc:/host/proc \
           -v /sys:/host/sys   \
           -p 2021:2021        \
           fluent/fluent-bit:1.8.0 \
           /fluent-bit/bin/fluent-bit \
           -i node_exporter_metrics \
           -p path.procfs=/host/proc \
           -p path.sysfs=/host/sys \
           -o prometheus_exporter \
           -p "add_label=host $HOSTNAME" \
           -f 1

Monitoring

Learn how to monitor your Fluent Bit data pipelines

Fluent Bit includes features for monitoring the internals of your pipeline, in addition to connecting to Prometheus and Grafana, Health checks, and connectors to use external services:

HTTP server

Fluent Bit includes an HTTP server for querying internal information and monitoring metrics of each running plugin.

You can integrate the monitoring interface with Prometheus.

Get started

To get started, enable the HTTP server from the configuration file. The following configuration instructs Fluent Bit to start an HTTP server on TCP port 2020 and listen on all network interfaces:

Start Fluent Bit with the corresponding configuration chosen previously:

Fluent Bit starts and generates output in your terminal:

Use curl to gather information about the HTTP server. The following command sends the command output to the jq program, which outputs human-readable JSON data to the terminal.

REST API interface

Fluent Bit exposes the following endpoints for monitoring.

URI

Description

Data format

v1 metrics

The following descriptions apply to v1 metric endpoints.

`/api/v1/metrics/prometheus` endpoint

The following descriptions apply to metrics outputted in Prometheus format by the /api/v1/metrics/prometheus endpoint.

The following terms are key to understanding how Fluent Bit processes metrics:

Record: a single message collected from a source, such as a single long line in a file.
Chunk: log records ingested and stored by Fluent Bit input plugin instances. A batch of records in a chunk are tracked together as a single unit.
The Fluent Bit engine attempts to fit records into chunks of at most 2 MB, but the size can vary at runtime. Chunks are then sent to an output. An output plugin instance can successfully send the full chunk to the destination and mark it as successful, or it can fail the chunk entirely if an unrecoverable error is encountered, or it can ask for the chunk to be retried.

Metric name

Labels

Description

Type

Unit

`/api/v1/storage` endpoint

The following descriptions apply to metrics outputted in JSON format by the /api/v1/storage endpoint.

Metric Key

Description

Unit

v2 metrics

The following descriptions apply to v2 metric endpoints.

`/api/v2/metrics/prometheus` or `/api/v2/metrics` endpoint

The following descriptions apply to metrics outputted in Prometheus format by the /api/v2/metrics/prometheus or /api/v2/metrics endpoints.

The following terms are key to understanding how Fluent Bit processes metrics:

Record: a single message collected from a source, such as a single long line in a file.
Chunk: log records ingested and stored by Fluent Bit input plugin instances. A batch of records in a chunk are tracked together as a single unit.
The Fluent Bit engine attempts to fit records into chunks of at most 2 MB, but the size can vary at runtime. Chunks are then sent to an output. An output plugin instance can either successfully send the full chunk to the destination and mark it as successful, or it can fail the chunk entirely if an unrecoverable error is encountered, or it can ask for the chunk to be retried.

Metric Name

Labels

Description

Type

Unit

Storage layer

The following are detailed descriptions for the metrics collected by the storage layer.

Metric Name

Labels

Description

Type

Unit

Output latency metric

Introduced in Fluent Bit 4.0.6, the fluentbit_output_latency_seconds histogram metric captures end-to-end latency from the time a chunk is created by an input plugin until it's successfully delivered by an output plugin. This provides observability into chunk-level pipeline performance and helps identify slowdowns or bottlenecks in the output path.

Bucket configuration

The histogram uses the following default bucket boundaries, designed around the Fluent Bit typical flush interval of 1 second:

These boundaries provide:

High resolution around 1 s latency: Captures normal operation near the default flush interval.
Small backpressure detection: Identifies minor delays in the 1-2.5 s range.
Bottleneck identification: Detects retry cycles, network stalls, or plugin bottlenecks in higher ranges.
Complete coverage**: The +Inf bucket ensures all latencies are captured.

Example output

When exposed through the Fluent Bit built-in HTTP server, the metric appears in Prometheus format:

Use cases

Performance monitoring: Monitor overall pipeline health by tracking latency percentiles:

Bottleneck detection: Identify specific input/output pairs experiencing high latency:

SLA monitoring: Track how many chunks are delivered within acceptable time bounds:

Alerting: Create alerts for degraded pipeline performance:

Uptime example

Query the service uptime with the following command:

The command prints a similar output like this:

Metrics example

Query internal metrics in JSON format with the following command:

The command prints a similar output like this:

Query metrics in Prometheus format

Query internal metrics in Prometheus Text 0.0.4 format:

This command returns the same metrics in Prometheus format instead of JSON:

Configure aliases

By default, configured plugins on runtime get an internal name in the format _plugin_name.ID_. For monitoring purposes, this can be confusing if many plugins of the same type were configured. To make a distinction each configured input or output section can get an alias that will be used as the parent name for the metric.

The following example sets an alias to the INPUT section of the configuration file, which is using the input plugin:

When querying the related metrics, the aliases are returned instead of the plugin name:

Grafana dashboard and alerts

You can create Grafana dashboards and alerts using Fluent Bit exposed Prometheus style metrics.

The provided is heavily inspired by 's with a few key differences, such as the use of the instance label, stacked graphs, and a focus on Fluent Bit metrics. See for more information.

Alerts

Sample alerts .

Health check for Fluent Bit

Fluent Bit supports the following configurations to set up the health check.

Configuration name

Description

Default

Not every error log means an error to be counted. The error retry failures count only on specific errors, which is the example in configuration table description.

Based on the HC_Period setting, if the real error number is over HC_Errors_Count, or retry failure is over HC_Retry_Failure_Count, Fluent Bit is considered unhealthy. The health endpoint returns an HTTP status 500 and an error message. Otherwise, the endpoint returns HTTP status 200 and an ok message.

The equation to calculate this behavior is:

The HC_Errors_Count and HC_Retry_Failure_Count only count for output plugins and count a sum for errors and retry failures from all running output plugins.

The following configuration examples show how to define these settings:

Use the following command to call the health endpoint:

With the example configuration, the health status is determined by the following equation:

If this equation evaluates to TRUE, then Fluent Bit is unhealthy.
If this equation evaluates to FALSE, then Fluent Bit is healthy.

Telemetry Pipeline

is a hosted service that lets you monitor your Fluent Bit agents including data flow, metrics, and configurations.

-- The C compiler identification is GNU 4.9.2
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- The CXX compiler identification is GNU 4.9.2
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
...
-- Could NOT find Doxygen (missing:  DOXYGEN_EXECUTABLE)
-- Looking for accept4
-- Looking for accept4 - not found
-- Configuring done
-- Generating done
-- Build files have been written to: /home/edsiper/coding/fluent-bit/build

Scanning dependencies of target msgpack
[  2%] Building C object lib/msgpack-1.1.0/CMakeFiles/msgpack.dir/src/unpack.c.o
[  4%] Building C object lib/msgpack-1.1.0/CMakeFiles/msgpack.dir/src/objectc.c.o
[  7%] Building C object lib/msgpack-1.1.0/CMakeFiles/msgpack.dir/src/version.c.o
...
[ 19%] Building C object lib/monkey/mk_core/CMakeFiles/mk_core.dir/mk_file.c.o
[ 21%] Building C object lib/monkey/mk_core/CMakeFiles/mk_core.dir/mk_rconf.c.o
[ 23%] Building C object lib/monkey/mk_core/CMakeFiles/mk_core.dir/mk_string.c.o
...
Scanning dependencies of target fluent-bit-static
[ 66%] Building C object src/CMakeFiles/fluent-bit-static.dir/flb_pack.c.o
[ 69%] Building C object src/CMakeFiles/fluent-bit-static.dir/flb_input.c.o
[ 71%] Building C object src/CMakeFiles/fluent-bit-static.dir/flb_output.c.o
...
Linking C executable ../bin/fluent-bit
[100%] Built target fluent-bit-bin

$ curl -s http://127.0.0.1:2020 | jq
{
  "fluent-bit": {
    "version": "0.13.0",
    "edition": "Community",
    "flags": [
      "FLB_HAVE_TLS",
      "FLB_HAVE_METRICS",
      "FLB_HAVE_SQLDB",
      "FLB_HAVE_TRACE",
      "FLB_HAVE_HTTP_SERVER",
      "FLB_HAVE_FLUSH_LIBCO",
      "FLB_HAVE_SYSTEMD",
      "FLB_HAVE_VALGRIND",
      "FLB_HAVE_FORK",
      "FLB_HAVE_PROXY_GO",
      "FLB_HAVE_REGEX",
      "FLB_HAVE_C_TLS",
      "FLB_HAVE_SETJMP",
      "FLB_HAVE_ACCEPT4",
      "FLB_HAVE_INOTIFY"
    ]
  }
}

# HELP fluentbit_output_latency_seconds End-to-end latency in seconds
# TYPE fluentbit_output_latency_seconds histogram
fluentbit_output_latency_seconds_bucket{le="0.5",input="random.0",output="stdout.0"} 0
fluentbit_output_latency_seconds_bucket{le="1.0",input="random.0",output="stdout.0"} 1
fluentbit_output_latency_seconds_bucket{le="1.5",input="random.0",output="stdout.0"} 6
fluentbit_output_latency_seconds_bucket{le="2.5",input="random.0",output="stdout.0"} 6
fluentbit_output_latency_seconds_bucket{le="5.0",input="random.0",output="stdout.0"} 6
fluentbit_output_latency_seconds_bucket{le="10.0",input="random.0",output="stdout.0"} 6
fluentbit_output_latency_seconds_bucket{le="20.0",input="random.0",output="stdout.0"} 6
fluentbit_output_latency_seconds_bucket{le="30.0",input="random.0",output="stdout.0"} 6
fluentbit_output_latency_seconds_bucket{le="+Inf",input="random.0",output="stdout.0"} 6
fluentbit_output_latency_seconds_sum{input="random.0",output="stdout.0"} 6.0015411376953125
fluentbit_output_latency_seconds_count{input="random.0",output="stdout.0"} 6

# 95th percentile latency
histogram_quantile(0.95, rate(fluentbit_output_latency_seconds_bucket[5m]))

# Average latency
rate(fluentbit_output_latency_seconds_sum[5m]) / rate(fluentbit_output_latency_seconds_count[5m])

# Example Prometheus alerting rule
- alert: FluentBitHighLatency
  expr: histogram_quantile(0.95, rate(fluentbit_output_latency_seconds_bucket[5m])) > 5
  for: 2m
  labels:
    severity: warning
  annotations:
    summary: "Fluent Bit pipeline experiencing high latency"
    description: "95th percentile latency is {{ $value }}s for {{ $labels.input }} -> {{ $labels.output }}"

{
  "input": {
    "cpu.0": {
      "records": 8,
      "bytes": 2536
    }
  },
  "output": {
    "stdout.0": {
      "proc_records": 5,
      "proc_bytes": 1585,
      "errors": 0,
      "retries": 0,
      "retries_failed": 0
    }
  }
}

fluentbit_input_records_total{name="cpu.0"} 57 1509150350542
fluentbit_input_bytes_total{name="cpu.0"} 18069 1509150350542
fluentbit_output_proc_records_total{name="stdout.0"} 54 1509150350542
fluentbit_output_proc_bytes_total{name="stdout.0"} 17118 1509150350542
fluentbit_output_errors_total{name="stdout.0"} 0 1509150350542
fluentbit_output_retries_total{name="stdout.0"} 0 1509150350542
fluentbit_output_retries_failed_total{name="stdout.0"} 0 1509150350542

service:
  http_server: on
  http_listen: 0.0.0.0
  http_port: 2020

pipeline:
  inputs:
    - name: cpu
      alias: server1_cpu

  outputs:
    - name: stdout
      alias: raw_output
      match: '*'

{
  "input": {
    "server1_cpu": {
      "records": 8,
      "bytes": 2536
    }
  },
  "output": {
    "raw_output": {
      "proc_records": 5,
      "proc_bytes": 1585,
      "errors": 0,
      "retries": 0,
      "retries_failed": 0
    }
  }
}

service:
  http_server: on
  http_listen: 0.0.0.0
  http_port: 2020
  health_check: on
  hc_errors_count: 5
  hc_retry_failure_count: 5
  hc_period: 5

pipeline:
  inputs:
    - name: cpu

  outputs:
    - name: stdout
      match: '*'

[SERVICE]
  HTTP_Server  On
  HTTP_Listen  0.0.0.0
  HTTP_PORT    2020
  Health_Check On
  HC_Errors_Count 5
  HC_Retry_Failure_Count 5
  HC_Period 5

[INPUT]
  Name  cpu

[OUTPUT]
  Name  stdout
  Match *

Docker

Fluent Bit container images are available on Docker Hub ready for production usage. Current available images can be deployed in multiple architectures.

Start Docker

Use the following command to start Docker with Fluent Bit:

Use a configuration file

Use the following command to start Fluent Bit while using a configuration file:

Tags and versions

The following table describes the Linux container tags that are available on Docker Hub repository:

Multi-architecture images

Fluent Bit production stable images are based on . Focusing on security, these images contain only the Fluent Bit binary and minimal system libraries and basic configuration.

Debug images are available for all architectures (for 1.9.0 and later), and contain a full Debian shell and package manager that can be used to troubleshoot or for testing purposes.

From a deployment perspective, there's no need to specify an architecture. The container client tool that pulls the image gets the proper layer for the running architecture.

Verify signed container images

Version 1.9 and 2.0 container images are signed using Cosign/Sigstore. Verify these signatures using cosign ():

Replace cosign with the binary installed if it has a different name (for example, cosign-linux-amd64).

Keyless signing is also provided but is still experimental:

COSIGN_EXPERIMENTAL=1 is used to allow verification of images signed in keyless mode. To learn more about keyless signing, see the documentation.

Get started

Download the last stable image from 2.0 series:
After the image is in place, run the following test which makes Fluent Bit measure CPU usage by the container:

That command lets Fluent Bit measure CPU usage every second and flushes the results to the standard output. For example:

FAQ

Why there is no Fluent Bit Docker image based on Alpine Linux?

Alpine Linux uses Musl C library instead of Glibc. Musl isn't fully compatible with Glibc, which generated many issues in the following areas when used with Fluent Bit:

Memory Allocator: To run properly in high-load environments, Fluent Bit uses Jemalloc as a default memory allocator which reduces fragmentation and provides better performance. Jemalloc can't run smoothly with Musl and requires extra work.
Alpine Linux Musl functions bootstrap have a compatibility issue when loading Golang shared libraries. This causes problems when trying to load Golang output plugins in Fluent Bit.
Alpine Linux Musl Time format parser doesn't support Glibc extensions.
The Fluent Bit maintainers' preference for base images are Distroless and Debian for security and maintenance reasons.

Why use distroless containers?

The reasons for using distroless are well covered in

Include only what you need, reduce the attack surface available.
Reduces size and improves performance.
Reduces false positives on scans (and reduces resources required for scanning).
Reduces supply chain security requirements to only what you need.

With any choice, there are downsides:

No shell or package manager to update or add things.
- Generally, dynamic updating is a bad idea in containers as the time it's done affects the outcome: two containers started at different times using the same base image can perform differently or get different dependencies.
- A better approach is to rebuild a new image version. You can do this with Distroless, but it's harder and requires multistage builds or similar to provide the new dependencies.

Using exec to access a container will potentially impact resource limits.

For debugging, debug containers are available now in K8S:

This can be a significantly different container from the one you want to investigate, with lots of extra tools or even a different base.
No resource limits applied to this container, which can be good or bad.
Runs in pod namespaces. It's another container that can access everything the others can.
Might need architecture of the pod to share volumes or other information.

$ cosign verify --key "https://packages.fluentbit.io/fluentbit-cosign.pub" fluent/fluent-bit:2.0.6

Verification for index.docker.io/fluent/fluent-bit:2.0.6 --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - The signatures were verified against the specified public key

[{"critical":{"identity":{"docker-reference":"index.docker.io/fluent/fluent-bit"},"image":{"docker-manifest-digest":"sha256:c740f90b07f42823d4ecf4d5e168f32ffb4b8bcd87bc41df8f5e3d14e8272903"},"type":"cosign container image signature"},"optional":{"release":"2.0.6","repo":"fluent/fluent-bit","workflow":"Release from staging"}}]

[2019/10/01 12:29:02] [ info] [engine] started
[0] cpu.0: [1504290543.000487750, {"cpu_p"=>0.750000, "user_p"=>0.250000, "system_p"=>0.500000, "cpu0.p_cpu"=>0.000000, "cpu0.p_user"=>0.000000, "cpu0.p_system"=>0.000000, "cpu1.p_cpu"=>1.000000, "cpu1.p_user"=>0.000000, "cpu1.p_system"=>1.000000, "cpu2.p_cpu"=>1.000000, "cpu2.p_user"=>1.000000, "cpu2.p_system"=>0.000000, "cpu3.p_cpu"=>0.000000, "cpu3.p_user"=>0.000000, "cpu3.p_system"=>0.000000}]

4.2

About

Concepts

Installation

Administration

Local testing

Data pipeline

Fluent Bit documentation

Features

Release notes

Fluent Bit, Fluentd, and CNCF

License

What is Fluent Bit?

The history of Fluent Bit

Build from source code

Stable

Development

Next step

Lab resources

O11y workshops by Chronosphere

Build with static configuration

Get started

Fluentd and Fluent Bit

Debian

Ubuntu

Server GPG key

Server GPG key

Update your sources lists

Update your repositories database

Install Fluent Bit

Update your repositories database

Install Fluent Bit

Logging with Fluent Bit and Amazon OpenSearch workshop by Amazon

Configuration directory

Build with custom configuration

Rocky Linux and Alma Linux

RHEL 9

Configure YUM

Install

Classic configuration files

Inputs

Data pipeline

Inputs

Buildroot embedded Linux

Install

Yocto embedded Linux

YAML configuration files

Parsers

Syntax

Multiline parsers

Syntax

Plugins

Inline YAML

Upstream servers

Variables

Memory management

Estimating

Fluent Bit and other architectures

Support

Filter

Buffer

Routing

Output

Standalone parsers files

Add a standalone parsers file to Fluent Bit

Example

Glibc and memory fragmentation

Key concepts

Events or records

Event format

Filtering

Tag

Timestamp

Match

Structured messages

Download and install Fluent Bit

Build from source code

Supported platforms and packages

Container deployment

Linux

Entries: `key`/`value`

Reading files with `tail`