Fluent Bit’s official Helm chart is surprisingly flexible, letting you choose between running it as a DaemonSet (the default, one per node) or a Deployment (a single instance for centralized logging).
Let’s get Fluent Bit installed on Kubernetes using its official Helm chart. We’ll start with the default DaemonSet configuration, which is great for collecting logs from every node in your cluster.
First, you need to add the Fluent Bit Helm repository.
helm repo add fluentbit https://fluent.github.io/helm-charts
helm repo update
Now, let’s install Fluent Bit. We’ll give our release a name, my-fluentbit, and specify the namespace logging (which we’ll create if it doesn’t exist).
kubectl create namespace logging --dry-run=client -o yaml | kubectl apply -f -
helm install my-fluentbit fluentbit/fluent-bit --namespace logging
This command deploys Fluent Bit as a DaemonSet. You can verify its status:
kubectl get pods -n logging -l app.kubernetes.io/name=fluent-bit
You should see one Fluent Bit pod running on each of your Kubernetes nodes. These pods are configured by default to tail logs from /var/log/containers/*.log on the host and send them to stdout. This means they’re ready to be picked up by another logging aggregator or to be viewed directly if you want to see raw logs from your nodes.
The real power comes from its configuration. The Helm chart allows extensive customization through its values.yaml file. You can override these defaults by passing --set flags or by creating your own values.yaml file and using helm install -f my-values.yaml.
Let’s say you want to send your logs to Elasticsearch. You’d typically modify the Fluent Bit configuration to include an Elasticsearch output plugin. This is done within the config.service.pipelines section of the Helm chart’s values.
Here’s a snippet of how you might configure an Elasticsearch output in your my-values.yaml:
config:
service:
pipelines:
metrics: "kubernetes | kubernetes_audit | stdout"
inputs: |
[INPUT]
Name tail
Path /var/log/containers/*.log
Parser docker
Tag kube.*
Mem_Buf_Limit 5MB
Skip_Long_Lines On
Refresh_Interval 1
outputs: |
[OUTPUT]
Name es
Match kube.*
Host elasticsearch.logging.svc.cluster.local
Port 9200
Logstash_Format On
Replace_Dots On
Retry_Limit False
And then you would install it like this:
helm install my-fluentbit fluentbit/fluent-bit --namespace logging -f my-values.yaml
This configuration tells Fluent Bit to:
- Use the
tailinput plugin to read container logs. - Parse these logs using the
dockerparser. - Tag logs matching
kube.*. - Send these tagged logs to an Elasticsearch instance running at
elasticsearch.logging.svc.cluster.localon port9200. - Format the logs for Logstash compatibility and replace dots in field names.
You can also change Fluent Bit’s deployment strategy. If you want a single, centralized Fluent Bit instance instead of one per node, you can switch from a DaemonSet to a Deployment. This is useful for scenarios where you have a dedicated log aggregation service or want to reduce resource overhead on your worker nodes.
To do this, you’d modify the service.type and daemonSet.enabled values:
service:
type: Deployment
daemonSet:
enabled: false
deployment:
replicas: 1
And install with:
helm install my-fluentbit fluentbit/fluent-bit --namespace logging -f my-values.yaml
This creates a single Fluent Bit pod managed by a Deployment. You’ll need to ensure this single pod has access to all the logs it needs to collect, potentially by using hostPath volumes or other mechanisms depending on your cluster’s setup and how logs are stored.
The Helm chart also offers fine-grained control over parsing, filtering, and buffering. For instance, you can define custom parsers or enable specific filters to enrich your log data before it’s sent to its destination. The fluentbit.conf file within the chart’s values.yaml is where you’ll spend most of your time customizing the behavior.
A common, yet often overlooked, configuration detail involves the mem_buf_limit. If your nodes generate a high volume of logs, or if your output destination experiences temporary network issues, this buffer can fill up. By default, it’s set to 5MB. If you see logs getting dropped or Fluent Bit restarting with memory pressure, increasing this value is a primary suspect. You’d add it to your [INPUT] section in values.yaml:
config:
inputs: |
[INPUT]
Name tail
Path /var/log/containers/*.log
Parser docker
Tag kube.*
Mem_Buf_Limit 100MB # Increased buffer limit
Skip_Long_Lines On
Refresh_Interval 1
This tells the tail input plugin to allocate up to 100MB of memory for buffering logs before they are processed and sent to an output. This can prevent data loss during transient network glitches or when the downstream system is temporarily overloaded.
Finally, after installing or upgrading, the next immediate hurdle you’ll likely face is ensuring your chosen output destination (like Elasticsearch, Splunk, or Loki) is correctly configured to receive and index the logs Fluent Bit is sending.