Vector and Fluent Bit are both popular choices for shipping logs to Loki, but they approach the task with different philosophies and offer distinct advantages.
Here’s Loki receiving logs from Vector, which is itself receiving logs from Fluent Bit:
# Fluent Bit config (fluent-bit.conf)
[SERVICE]
Flush 5
Daemon off
Log_Level info
Parsers_File parsers.conf
[INPUT]
Name tail
Path /var/log/app/*.log
Tag app.*
Parser json
[OUTPUT]
Name forward
Host 127.0.0.1
Port 24224
Format json
# Vector config (vector.toml)
[sources.fluentbit_input]
type = "tcp"
address = "127.0.0.1:24224"
mode = "server"
[transforms.parse_json]
type = "json_parser"
inputs = ["fluentbit_input"]
[sinks.loki_sink]
type = "loki"
endpoint = "http://localhost:3100/loki/api/v1/push"
inputs = ["parse_json"]
encoding = "json"
labels = { namespace = "default", app = "{{ .filename | replace_all \"/\" \"-\" }}" }
In this setup, Fluent Bit tails log files, parses them as JSON, and forwards them over TCP to Vector. Vector then listens on TCP, receives the JSON data, parses it again (ensuring it’s valid JSON), and sends it to Loki with specific labels derived from the original filename. The namespace label is static, while the app label dynamically extracts the filename from the log source and sanitizes it for Loki.
The core problem both Vector and Fluent Bit solve is the reliable, efficient, and structured collection and forwarding of logs from diverse sources to a centralized logging system like Loki. They act as agents that can be deployed on application hosts, Kubernetes nodes, or even as sidecars, abstracting away the complexities of direct file access, network buffering, retries, and transformation.
Internally, both use a pipeline model. Fluent Bit’s pipeline is configured via INPUT, FILTER, and OUTPUT plugins. Vector uses sources, transforms, and sinks. This allows for a modular approach where you can chain together different stages to achieve your desired log processing. For example, you could add a grep filter in Fluent Bit to drop certain log lines before they even reach Vector, or a remap transform in Vector to enrich log events with additional metadata before sending them to Loki.
The labels configuration in the Vector sink is crucial for Loki’s query performance and organization. Without meaningful labels, searching through logs becomes a slow, full-text scan. By default, Loki uses labels like filename and host if you’re using the promtail agent. However, with Vector, you have fine-grained control. Here, we’re explicitly setting namespace and deriving app from the log source filename. This allows you to quickly filter logs by namespace="default" and app="var-log-app-myapp.log" (after the replacement), which is far more efficient than searching for the string var/log/app/myapp.log within the log content itself.
What many folks don’t realize is that Vector’s json_parser transform, when applied to data received from an tcp source that’s acting as a server, will happily parse each line of the incoming TCP stream as a separate JSON object. This means if Fluent Bit sends multiple JSON objects separated by newlines, Vector will correctly interpret each as an independent log event, preserving the structure and meaning of the original log message. It doesn’t treat the entire TCP stream as a single blob of text to be parsed once.
The next logical step is to explore how to handle log rotation and ensure no log data is lost during agent restarts or network interruptions.