Fluentd’s HTTP webhook input is surprisingly flexible, allowing you to ingest events not just as simple JSON but as arbitrary HTTP request bodies, which can be a game-changer for integrating with systems that don’t naturally produce JSON.
Let’s see this in action. Imagine you have a simple Python Flask app that’s tracking user actions and wants to send them to Fluentd. Instead of formatting each event as JSON, it just sends the raw action string.
from flask import Flask, request
import requests
app = Flask(__name__)
@app.route('/log_action', methods=['POST'])
def log_action():
action_data = request.data.decode('utf-8') # Get raw body
webhook_url = "http://fluentd-host:9880/webhook.log" # Fluentd HTTP input port
try:
response = requests.post(webhook_url, data=action_data)
if response.status_code == 200:
return "Event logged successfully", 200
else:
return f"Error logging event: {response.status_code}", 500
except requests.exceptions.RequestException as e:
return f"Error sending to Fluentd: {e}", 500
if __name__ == '__main__':
app.run(port=5000)
Now, on the Fluentd side, you need a configuration that can handle this raw data. This is where the http input plugin shines, specifically with its format none option.
<source>
@type http
port 9880
bind 0.0.0.0
<parse>
@type none # Crucial for raw body ingestion
</parse>
tag http.webhook.raw_events
</source>
<match http.webhook.raw_events>
@type stdout # Or your preferred output plugin (e.g., file, elasticsearch)
</match>
When your Flask app sends a POST request to http://fluentd-host:9880/webhook.log with a body like user_logged_in, Fluentd receives this raw string. Because format none is specified, Fluentd doesn’t try to parse the body into structured fields. Instead, it treats the entire raw request body as the event’s message. The tag http.webhook.raw_events is applied, and this raw string will be outputted (or forwarded to your destination).
The problem this solves is integrating with legacy systems or microservices that emit simple text-based logs or event payloads rather than structured JSON. You can avoid the overhead of building custom parsers on the sending side; Fluentd just accepts what it’s given.
Internally, when format none is used, the http input plugin simply takes the request.body from the incoming HTTP request and assigns it to the message field of the Fluentd event record. No further processing or field extraction occurs within the input plugin itself. The tag specified in the <source> block is directly applied to this raw event.
The exact levers you control are the port and bind for listening, the tag for routing, and crucially, the <parse> block. Setting @type none within <parse> is the key to treating the entire request body as the event’s payload. If you were to omit <parse> entirely, Fluentd would default to trying to parse the body as JSON, which would likely fail for non-JSON data and result in errors.
What most people don’t realize is that the http input can also accept form-encoded data (application/x-www-form-urlencoded) when format none is used. In this scenario, the entire query string or form body is treated as the raw message. For example, a POST request with body user_id=123&action=view_page&page=/home would result in a single event where the message field contains the literal string user_id=123&action=view_page&page=/home. This is distinct from how JSON or other parsed formats would handle form data, where individual fields would be extracted.
If you later decide you do want to parse the raw body into structured fields, you’d change the <parse> block to something like <parse @type json> or <parse @type regexp ...>.