Flatten and Nest JSON Fields in Fluent Bit with Nest Filter (2026)

The nest filter in Fluent Bit can dramatically simplify your log processing by restructuring complex JSON into a more manageable, hierarchical format.

Let’s see it in action. Imagine you have logs like this, with deeply nested fields you want to group:

{
  "timestamp": "2023-10-27T10:00:00Z",
  "level": "info",
  "message": "User logged in",
  "user": {
    "id": "12345",
    "username": "alice",
    "profile": {
      "name": "Alice Smith",
      "email": "alice@example.com"
    }
  },
  "request": {
    "method": "GET",
    "url": "/api/v1/users/12345",
    "headers": {
      "User-Agent": "Mozilla/5.0",
      "X-Request-ID": "abc-123"
    }
  }
}

You want to flatten the user.profile fields and nest the user and request data under new top-level keys. You can achieve this with a nest filter configuration like this:

[SERVICE]
    Flush        5
    Daemon       off
    Log_Level    info

[INPUT]
    Name         dummy
    Tag          dummy

[FILTER]
    Name         nest
    Match        dummy
    Operation    nest
    Nested_Key   user_data
    Add_Keys     user.id,user.username,user.profile.name,user.profile.email

[FILTER]
    Name         nest
    Match        dummy
    Operation    nest
    Nested_Key   request_data
    Add_Keys     request.method,request.url,request.headers.User-Agent,request.headers.X-Request-ID

[OUTPUT]
    Name         stdout
    Match        dummy
    Format       json

When Fluent Bit processes these logs with the above configuration, the output will look like this:

{
  "timestamp": "2023-10-27T10:00:00Z",
  "level": "info",
  "message": "User logged in",
  "user_data": {
    "id": "12345",
    "username": "alice",
    "name": "Alice Smith",
    "email": "alice@example.com"
  },
  "request_data": {
    "method": "GET",
    "url": "/api/v1/users/12345",
    "User-Agent": "Mozilla/5.0",
    "X-Request-ID": "abc-123"
  }
}

Notice how user.id, user.username, and the fields from user.profile are now under user_data. Similarly, the request fields are grouped under request_data. The original user and request top-level keys are gone, effectively flattening the specified nested paths and creating new, cleaner structures.

The nest filter’s Operation can be either nest or flatten. When nest is used, it takes the specified Add_Keys (which are dot-notation paths to existing fields) and moves them into a new top-level key defined by Nested_Key. The original fields are removed from their original locations. If you use flatten, it does the opposite: it takes fields from a specified Nested_Key and "promotes" them to the top level, removing them from the nested structure.

This filter is invaluable for normalizing disparate log formats. If some logs have user info at user.details.id and others at user.id, you can use multiple nest filters to bring them all to a consistent user_data.id field, making downstream analysis much simpler. The Add_Keys parameter accepts a comma-separated list of dot-notation paths. You can specify any level of nesting, and the filter will recursively create the necessary intermediate JSON objects if they don’t exist.

The core problem this solves is the inherent complexity and inconsistency of nested JSON structures in logs. As applications evolve, log formats can become deeply nested and difficult to query. The nest filter allows you to impose a flatter, more predictable schema on your logs before they hit your analysis or storage systems. This means you can query for user_data.email consistently, regardless of whether the original log had it at user.profile.email or user.contact_info.email.

A common point of confusion is how the flatten operation works when you want to bring multiple nested fields to the top level. You’d typically use flatten with Nested_Key pointing to the structure you want to de-nest, and Add_Keys specifying which fields within that structure should be promoted. For instance, if you had {"user": {"id": "123", "name": "Bob"}} and wanted id and name at the top level, you’d use Operation flatten, Nested_Key user, and Add_Keys id,name. This would result in {"id": "123", "name": "Bob"} and the original user key would be removed.

The nest filter, by default, will create intermediate JSON objects if they don’t exist. So, if you specify Add_Keys user.profile.address and only user exists, it will create profile and address as empty JSON objects before placing the value. This behavior is usually desired but can be a subtle source of unexpected empty objects if not carefully managed.

The next concept you’ll likely encounter is how to handle conflicting field names when flattening or nesting, and the strategies for resolving them, often involving renaming fields in subsequent filter stages.