Lambda + Kinesis: Process Stream Records in Batches (2026)

Lambda functions, when triggered by Kinesis Data Streams, process records in batches, not one by one. This batching is key to efficiency and managing throughput.

Let’s see this in action. Imagine a Kinesis stream named my-data-stream. We have a Lambda function, kinesis-processor, configured to poll this stream. When data arrives, Kinesis doesn’t immediately invoke Lambda for each individual record. Instead, it accumulates records until a certain threshold is met or a timeout occurs. Then, it invokes the Lambda function with a payload containing an array of these records.

Here’s a simplified example of what the event object might look like for a Lambda function triggered by Kinesis:

{
  "Records": [
    {
      "kinesisSchemaVersion": "1.0",
      "partitionKey": "shardId-000000000000",
      "sequenceNumber": "49590338271490256608566034412747477696011298004109859329",
      "data": "eyAidGVtcGVyYXR1cmUiOiA3MCwgImh1bWlkaXR5IjogNTB9", // Base64 encoded JSON: { "temperature": 70, "humidity": 50 }
      "approximateArrivalTimestamp": 1545084650.987
    },
    {
      "kinesisSchemaVersion": "1.0",
      "partitionKey": "shardId-000000000000",
      "sequenceNumber": "49590338271490256608566034412747477696011298004109859330",
      "data": "eyAidGVtcGVyYXR1cmUiOiA3MiwgImh1bWlkaXR5IjogNTF9", // Base64 encoded JSON: { "temperature": 72, "humidity": 51 }
      "approximateArrivalTimestamp": 1545084651.123
    }
    // ... more records
  ]
}

The Records array is where your batch of data resides. Your Lambda function’s job is to iterate through this array, process each record, and then acknowledge successful processing.

The core problem this batching solves is reducing invocation overhead. If Lambda invoked your function for every single record, the network latency and computational cost of starting a Lambda execution environment for each record would quickly overwhelm your processing capacity and become prohibitively expensive. Batching amortizes this overhead across multiple records.

Internally, Kinesis Data Streams uses enhanced fan-out consumers (which Lambda leverages) to push data to Lambda. Lambda polls these shards. When it retrieves a batch of records, it invokes your function. The size of this batch is configurable. You can set BatchSize in your Lambda’s Kinesis trigger configuration to a value between 1 and 10,000 records. The MaximumBatchingWindowInSeconds setting (defaulting to 0, meaning no window) controls how long Lambda waits for records to accumulate before invoking the function, even if the BatchSize isn’t met. A non-zero value, like 60 seconds, means Lambda will wait up to 60 seconds for more records to arrive before invoking, if the batch size hasn’t been reached.

To process these records, your Lambda function code typically looks like this (Python example):

import base64
import json

def lambda_handler(event, context):
    processed_records = []
    for record in event['Records']:
        payload = base64.b64decode(record['data']).decode('utf-8')
        data = json.loads(payload)

        # --- Your processing logic here ---
        print(f"Processing record: {data}")
        processed_data = {
            "temperature": data.get("temperature", 0) * 1.8 + 32, # Convert C to F
            "humidity": data.get("humidity", 0)
        }
        processed_records.append(processed_data)
        # ---------------------------------

    # If all records in the batch were processed successfully,
    # Kinesis will advance the sequence number for this shard.
    # If an exception is raised, the batch will be retried.
    print(f"Successfully processed {len(processed_records)} records.")
    return {
        'statusCode': 200,
        'body': json.dumps('Batch processed successfully!')
    }

The most critical aspect of handling Kinesis batches in Lambda is error handling and idempotency. If your Lambda function throws an unhandled exception, the entire batch of records will be retried by Kinesis. This means your function must be able to handle the same records multiple times without causing duplicate side effects (e.g., writing the same data twice to a database). You achieve this by designing your processing logic to be idempotent. For example, if you’re writing to a database, use unique identifiers and UPSERT operations rather than simple INSERTs.

When you configure your Lambda function’s Kinesis trigger, you can set BisectBatchOnFunctionError. If set to true, and your function fails, Lambda will attempt to re-invoke your function with smaller batches, starting from the record that caused the failure, isolating the problematic record. This helps pinpoint errors but can lead to slower recovery if the issue isn’t in a single record.

The next concept you’ll grapple with is managing the sequence numbers. Kinesis uses sequence numbers to track which records have been successfully processed. When your Lambda function successfully returns, Lambda informs Kinesis, which then advances the sequence number pointer for that shard. If your function fails, Kinesis keeps the pointer at the same position, meaning the batch will be re-processed on the next invocation. This is why idempotency is paramount – retries are a feature, not a bug.

The final piece of the puzzle is understanding the interaction between BatchSize and MaximumBatchingWindowInSeconds. If BatchSize is 100 and MaximumBatchingWindowInSeconds is 60, Lambda will invoke your function if either 100 records have arrived or if 60 seconds have passed, whichever comes first. If the stream is very low-traffic, you might get smaller batches more frequently due to the timeout, while high-traffic streams will likely hit the BatchSize limit often.