Lambda functions choke on payloads larger than 6MB when invoked synchronously, and 256KB when invoked asynchronously.

Here’s how to handle these constraints and keep your serverless functions from failing.

The Problem: Payload Size Limits

AWS Lambda has two primary payload size limits:

  1. Synchronous Invocations: When you invoke a Lambda function and wait for its response (e.g., via API Gateway, lambda.invoke with InvocationType: RequestResponse), the maximum payload size for both the request and the response is 6MB.
  2. Asynchronous Invocations: When you invoke a Lambda function and don’t wait for a result (e.g., via SQS, SNS, S3 events, lambda.invoke with InvocationType: Event), the maximum payload size for the request is 256KB.

Exceeding these limits results in errors. For synchronous calls, you’ll typically see a 413 Request Entity Too Large from API Gateway or a Lambda.MaxPayloadExceededException if invoking directly. For asynchronous calls, the event source might retry, or you’ll see errors indicating the payload was too large.

Why 6MB and 256KB?

These limits are rooted in how Lambda handles function invocations and manages resources. For synchronous calls, Lambda needs to buffer the entire request and response within its execution environment and return it promptly. A larger payload increases latency and memory usage. For asynchronous calls, the event sources (like SQS or SNS) have their own payload size limits that Lambda often aligns with or is constrained by, and Lambda also needs to efficiently queue and deliver these events.

Strategies to Handle Large Payloads

The core idea is to avoid putting large data directly into the Lambda invocation payload. Instead, use an intermediary storage service.

1. Storing Data in S3

This is the most common and robust solution for payloads exceeding the limits.

Scenario: Your Lambda function needs to process a large dataset (e.g., a CSV file, a JSON document, images) that’s too big for a direct invocation.

How it works:

  1. Upload to S3: The client or upstream service uploads the large data directly to an S3 bucket.
  2. Invoke Lambda with S3 Object Reference: The client then invokes the Lambda function, passing only the S3 object’s bucket name and key as the payload.
  3. Lambda Reads from S3: The Lambda function uses the provided bucket and key to retrieve the data from S3.

Example (Python):

  • Payload sent to Lambda:
    {
      "bucket": "my-large-data-bucket",
      "key": "uploads/large_document.json"
    }
    
  • Lambda Function Code:
    import boto3
    import json
    
    s3 = boto3.client('s3')
    
    def lambda_handler(event, context):
        bucket_name = event['bucket']
        object_key = event['key']
    
        try:
            # Get the object from S3
            response = s3.get_object(Bucket=bucket_name, Key=object_key)
            data = response['Body'].read().decode('utf-8')
    
            # Process the data (e.g., parse JSON, read CSV)
            # For this example, let's assume it's JSON
            processed_data = json.loads(data)
            print(f"Successfully retrieved and parsed data from {object_key}")
            # ... your processing logic here ...
    
            return {
                'statusCode': 200,
                'body': json.dumps('Data processed successfully!')
            }
        except Exception as e:
            print(f"Error processing object {object_key}: {e}")
            return {
                'statusCode': 500,
                'body': json.dumps(f'Error processing data: {str(e)}')
            }
    

Why it works: The Lambda payload is now tiny (just two strings), well within the 6MB or 256KB limit. S3 is designed for massive object storage, and Lambda can efficiently download objects of any size (within its own execution environment’s memory and timeout limits).

Key Considerations:

  • Permissions: Ensure your Lambda function’s IAM role has s3:GetObject permission for the relevant S3 bucket.
  • S3 Upload: The upstream service needs to handle the S3 upload. For larger files, it can use S3’s multipart upload API.
  • Response Payload: If your Lambda function generates a large response, you’ll need to apply the same S3 strategy: write the response to an S3 object and return the S3 object reference.

2. Using SQS for Asynchronous Processing

When you have a large message to send to a Lambda function that’s triggered asynchronously, SQS is an excellent intermediary.

Scenario: You need to queue up many processing tasks, and each task involves a substantial amount of data.

How it works:

  1. Send to SQS: The upstream service sends a message to an SQS queue. If the message data exceeds SQS’s 256KB limit, you can use SQS’s Large Message Storage feature. This feature automatically stores messages larger than 256KB in an S3 bucket and sends a reference (S3 object key and bucket) to the SQS queue.
  2. SQS Triggers Lambda: Configure an SQS trigger for your Lambda function.
  3. Lambda Reads from S3 (if necessary): If the SQS message payload is small, Lambda receives it directly. If it’s a large message, the payload will contain the S3 reference, and your Lambda function will retrieve the data from S3.

Example (SQS Message Payload - Large Message Storage):

{
  "s3": {
    "bucket": {
      "name": "my-sqs-large-message-bucket"
    },
    "object": {
      "key": "sqs-large-messages/message-xyz123.json"
    }
  }
}

Lambda Function Code (similar to S3 example):

import boto3
import json

s3 = boto3.client('s3')

def lambda_handler(event, context):
    for record in event['Records']:
        try:
            # Check if it's a large message stored in S3
            if 's3' in record['body']:
                message_body = json.loads(record['body'])
                s3_info = message_body['s3']
                bucket_name = s3_info['bucket']['name']
                object_key = s3_info['object']['key']

                print(f"Retrieving large message from S3: bucket={bucket_name}, key={object_key}")
                response = s3.get_object(Bucket=bucket_name, Key=object_key)
                data = response['Body'].read().decode('utf-8')
                # Process the actual large data
                actual_payload_data = json.loads(data) # Assuming it's JSON
                print("Successfully retrieved and parsed large message data.")
                # ... your processing logic ...
            else:
                # Process a regular, small SQS message
                payload_data = json.loads(record['body'])
                print("Processing regular SQS message.")
                # ... your processing logic ...

            # If successful, SQS will automatically delete the message.
            # If an exception is raised, SQS will retry based on visibility timeout.

        except Exception as e:
            print(f"Error processing SQS record: {e}")
            # Raising an exception will cause SQS to make the message visible again for retry.
            raise e
    return {
        'statusCode': 200,
        'body': json.dumps('SQS messages processed.')
    }

Why it works: SQS handles the complexity of large message storage. Lambda receives a small reference, and your function can then fetch the full data from S3. This keeps asynchronous invocations within the 256KB limit.

Key Considerations:

  • SQS Queue Configuration: You need to enable "Large message storage" on your SQS queue and specify an S3 bucket.
  • Lambda Trigger Configuration: Ensure the Lambda trigger is set up correctly for your SQS queue. The event structure will be an SQS ReceiveMessage response, with each message containing a body.
  • Error Handling: Proper error handling in Lambda is crucial for SQS. If your function fails to process a message, raising an exception will cause SQS to retry. If it succeeds, the message is deleted.

3. Using STS for Temporary Credentials with S3

When your Lambda needs to write large results back to S3, it will need permissions to do so. The standard way is via IAM roles. However, if you have a complex multi-account or cross-role scenario, you might temporarily assume a role using AWS Security Token Service (STS) to get temporary credentials for S3 operations.

Scenario: Your Lambda function processes data and needs to save the large output artifact to an S3 bucket in a different account, or with specific, temporary permissions.

How it works:

  1. Assume Role: Your Lambda function uses boto3.client('sts') to call assume_role with the ARN of the target role.
  2. Get Temporary Credentials: STS returns temporary access keys, secret access keys, and session tokens.
  3. Create S3 Client with Temp Credentials: Instantiate a new boto3.client('s3') using these temporary credentials.
  4. Upload to S3: Use this S3 client to upload the large result data to the target S3 bucket.

Example (Python):

import boto3
import json

def lambda_handler(event, context):
    # Assume the role in the target account
    sts_client = boto3.client('sts')
    try:
        assumed_role_object = sts_client.assume_role(
            RoleArn="arn:aws:iam::111122223333:role/TargetS3WriteRole",
            RoleSessionName="MyLambdaSession"
        )
        credentials = assumed_role_object['Credentials']

        # Create an S3 client with the temporary credentials
        s3_client_temp = boto3.client(
            's3',
            aws_access_key_id=credentials['AccessKeyId'],
            aws_secret_access_key=credentials['SecretAccessKey'],
            aws_session_token=credentials['SessionToken'],
        )

        # Prepare your large data (e.g., from processing)
        large_output_data = {"result": "...", "details": "..."} # This could be huge
        output_json_string = json.dumps(large_output_data)

        # Upload the data to S3 in the target account
        target_bucket = "my-target-results-bucket"
        target_key = "results/processed_output.json"

        s3_client_temp.put_object(
            Bucket=target_bucket,
            Key=target_key,
            Body=output_json_string.encode('utf-8')
        )
        print(f"Successfully wrote large output to s3://{target_bucket}/{target_key}")

        return {
            'statusCode': 200,
            'body': json.dumps('Large output saved to S3.')
        }

    except Exception as e:
        print(f"Error assuming role or writing to S3: {e}")
        return {
            'statusCode': 500,
            'body': json.dumps(f'Error: {str(e)}')
        }

Why it works: By assuming a role and using temporary credentials, your Lambda function can perform actions in other AWS accounts or with granular, time-limited permissions. The actual large data is transferred directly between the Lambda execution environment and S3, bypassing the Lambda invocation payload limits.

Key Considerations:

  • Trust Policy: The target role (TargetS3WriteRole in the example) must have a trust policy that allows your Lambda function’s execution role to assume it.
  • IAM Permissions: The target role must have s3:PutObject (or similar) permissions for the target bucket.
  • Session Duration: Temporary credentials have a limited lifespan (default 1 hour, max 12 hours).

The Next Hurdle: Lambda Timeout

Once you’ve successfully managed your payload sizes by offloading data to S3, your next challenge will likely be ensuring your Lambda function has enough time to process that data. The default Lambda timeout is 3 seconds, and the maximum is 15 minutes. For very large datasets, you might need to configure a longer timeout or break down the processing into smaller, parallelizable steps.

Want structured learning?

Take the full Lambda course →