Fluent Bit can export logs to OpenTelemetry Collectors via the OTLP protocol, but it’s not always straightforward to get the data structured correctly for analysis.
Here’s a typical setup: Fluent Bit running as a daemonset on Kubernetes, collecting container logs. The goal is to send these logs to an OpenTelemetry Collector (OTel Collector) that’s also running on Kubernetes, and then have that collector process and export them to a backend like Jaeger or Prometheus.
# Fluent Bit DaemonSet configuration snippet
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: fluent-bit
spec:
template:
spec:
containers:
- name: fluent-bit
image: fluent/fluent-bit:latest
volumeMounts:
- name: fluent-bit-config
mountPath: /fluent-bit/etc/
volumes:
- name: fluent-bit-config
configMap:
name: fluent-bit-config
# Fluent Bit configuration file (fluent-bit.conf)
[SERVICE]
Flush 5
Daemon On
Log_Level info
Parsers_File parsers.conf
[INPUT]
Name tail
Path /var/log/containers/*log
Parser docker
Tag kube.*
Mem_Buf_Limit 5MB
Skip_Long_Lines On
Refresh_Interval 10
[OUTPUT]
Name forward
Match kube.*
Host otel-collector.observability.svc.cluster.local
Port 4317
tls On
tls.verify Off
# OTel Collector configuration snippet (receivers, processors, exporters)
apiVersion: v1
kind: ConfigMap
metadata:
name: otel-collector-config
data:
config.yaml: |
receivers:
otlp:
protocols:
grpc:
http:
processors:
batch:
memory_limiter:
check_interval: 1s
limit_percentage: 75
spike_limit_percentage: 25
exporters:
logging:
loglevel: debug
# Example exporter to Jaeger
# jaeger:
# endpoint: jaeger-collector.observability.svc.cluster.local:14250
service:
pipelines:
logs:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [logging] # or [jaeger]
The core of sending logs from Fluent Bit to an OTel Collector using OTLP is the [OUTPUT] plugin in Fluent Bit. You’ll typically use the forward output plugin, which is designed to send data to another Fluent Bit instance or a compatible receiver like the OTel Collector’s OTLP receiver.
The Host and Port in the forward output plugin must point to your OTel Collector’s listening address. For OTLP over gRPC, this is usually port 4317. If you’re using OTLP over HTTP, it’s typically port 4318. Ensure your OTel Collector is configured with an otlp receiver listening on the correct protocol and port.
The Tag in the [INPUT] section acts as a routing mechanism within Fluent Bit. The Match in the [OUTPUT] section then selects which tags are sent to which output. Match kube.* means any log tag starting with kube. will be sent to the configured output.
When configuring the OTel Collector, the otlp receiver needs to be enabled for both grpc and http if you want to support both. The service section defines the pipelines. For logs, you specify the logs pipeline, linking the otlp receiver to your chosen processors and exporters.
A common stumbling block is the log format. Fluent Bit, by default, might send logs as plain text. For the OTel Collector to properly parse and understand these logs, especially for structured logging, you need to ensure Fluent Bit is parsing the log into a structured format (like JSON) before sending it. This is often done using the Parser directive in the [INPUT] section, pointing to a custom parser that can extract fields from your log lines.
For instance, if your application logs JSON, you’d have a parser like this in parsers.conf:
[PARSER]
Name json
Format json
Key_Content log
And in your Fluent Bit config:
[INPUT]
Name tail
Path /var/log/containers/*log
Parser json # Use the custom json parser
Tag kube.*
Mem_Buf_Limit 5MB
Skip_Long_Lines On
Refresh_Interval 10
This tells Fluent Bit to parse the log line as JSON. The OTel Collector’s OTLP receiver will then receive these structured logs, and if you’ve configured appropriate processors and exporters, they’ll be sent to your backend in a usable format.
If you’re seeing logs arrive at the OTel Collector but they aren’t structured as you expect, double-check your Fluent Bit parsers. The docker parser is common for container logs, but it might not always produce the granular structure you need. Consider using a more specific parser for your application’s log format.
The OTel Collector itself can also be configured to parse incoming data. However, it’s generally more efficient and predictable to have Fluent Bit do the initial parsing. If you’re using the logging exporter in the OTel Collector, you’ll see the raw data it receives, which is invaluable for debugging.
The tls.verify Off in the Fluent Bit output is a common shortcut for development/testing. In production, you’ll want to configure TLS certificates for secure communication between Fluent Bit and the OTel Collector.
Finally, remember that the OTel Collector can receive data from multiple sources and send it to multiple destinations. You might have Fluent Bit sending logs, and Prometheus scraping metrics, all processed and exported by the same OTel Collector instance.
If your logs are arriving at the OTel Collector but not appearing in your backend system, the next error you’ll likely encounter is a misconfiguration in the OTel Collector’s exporters section or a connectivity issue between the OTel Collector and your logging backend.