Lambda functions can poll an SQS queue and process messages automatically, but only if you configure the event source mapping correctly.
Let’s see this in action. Imagine you have an SQS queue named my-processing-queue and a Lambda function message-processor. When a message lands in my-processing-queue, we want message-processor to automatically pick it up and run.
Here’s a simplified Lambda function in Python that would process a message:
import json
import boto3
def lambda_handler(event, context):
for record in event['Records']:
message_body = record['body']
print(f"Processing message: {message_body}")
# Your actual message processing logic goes here
# For example, if the message is a JSON string:
try:
data = json.loads(message_body)
print(f"Parsed data: {data}")
# Process the parsed data
except json.JSONDecodeError:
print("Message is not valid JSON, processing as plain text.")
# Process as plain text
return {
'statusCode': 200,
'body': json.dumps('Messages processed successfully!')
}
To make this happen automatically, you need to create an SQS event source mapping. This mapping tells Lambda to poll your SQS queue and invoke your function with the messages it receives.
You can set this up in the AWS console under your Lambda function’s configuration, in the "Add trigger" section. Select "SQS" as the trigger type, choose your my-processing-queue, and then configure the settings.
Here are the key settings you’ll encounter and what they mean:
- Batch Size: This determines how many messages Lambda will fetch from the SQS queue in a single batch to send to your function. The default is 10, but you can set it between 1 and 10,000. A larger batch size can improve throughput if your function can process multiple messages efficiently, but it also increases the chance of partial batch failures.
- Batch Window: This is the maximum amount of time (in seconds) Lambda will wait to gather messages for a batch, up to the specified
Batch Size. If the batch size is reached before the window expires, Lambda sends the batch immediately. If the window expires and there are messages, Lambda sends whatever it has. This helps reduce latency for individual messages if your queue is not very busy. The default is 0, meaning it sends messages as soon as the batch size is met. - Starting Position: For a new event source mapping, you can specify whether Lambda should start processing from the
LATESTmessage in the queue orTRIM_HORIZON(the oldest message). If you’re re-enabling a disabled mapping, it defaults to processing messages from where it left off. - Enabled: A simple toggle to turn the event source mapping on or off.
Internally, Lambda creates a service-linked role that grants it permission to poll your SQS queue and invoke your function. The event source mapping is the bridge that connects the SQS queue to your Lambda function, orchestrating the polling and invocation process. Lambda polls the queue periodically. When it retrieves messages, it groups them into a batch (up to your configured Batch Size and within the Batch Window) and then invokes your Lambda function, passing the batch of messages in the event object.
Crucially, once a batch of messages is successfully processed by your Lambda function (meaning your function returns a 2xx status code, or doesn’t throw an unhandled exception), Lambda automatically deletes those messages from the SQS queue. If your function fails to process a batch, Lambda will not delete the messages, and they will become visible again in the queue after their visibility timeout expires, allowing for retries.
The most surprising thing about SQS event source mappings is their inherent retries and error handling. If your Lambda function fails to process a message (or a batch of messages) and returns an error, Lambda will retry invoking the function with that same batch. This retry mechanism is governed by the SQS queue’s maxReceiveCount and the Lambda event source mapping’s retry behavior. If the function consistently fails, after maxReceiveCount attempts, the message will be moved to the queue’s Dead-Letter Queue (DLQ) if one is configured. This automatic retry and DLQ mechanism is a powerful, often underestimated, feature for building resilient asynchronous processing systems.
Understanding the interplay between the SQS visibility timeout, the Lambda event source mapping’s batching behavior, and your function’s error handling is key to building robust, scalable message processing pipelines.
The next concept you’ll likely grapple with is how to effectively handle partial batch failures when your Lambda function processes a large batch of messages.