Skip to main content

Async Python in Lambda: Concurrent Invocations

Async/await in Python enables a single Lambda function to handle thousands of concurrent tasks efficiently using event loops instead of threads. Combined with Lambda's automatic scaling, you can build massively parallel workloads—image processing pipelines, API aggregation, batch jobs—without managing servers or thread pools.

What is Concurrency in Lambda?​

Concurrency is the number of Lambda invocations executing simultaneously. AWS automatically scales your function to handle traffic:

  • 1 request: 1 concurrent execution
  • 10 simultaneous requests: 10 concurrent executions (across 10 function instances or fewer with async)
  • 1,000 simultaneous requests: AWS auto-scales to 1,000 instances (each running independently) or fewer if using async internally

Lambda's default account concurrency limit is 1,000 across all functions. You can request an increase. Reserved concurrency guarantees a minimum; provisioned concurrency pre-warms instances.

Using Async/Await in Lambda Handlers​

Python's asyncio module enables concurrent I/O within a single function execution. This is different from Lambda's cross-invocation scaling; async handles multiple tasks within one invocation.

Example: Fetch multiple URLs concurrently

import asyncio
import aiohttp
import json

async def fetch_url(session, url):
    """Fetch a single URL asynchronously"""
    async with session.get(url) as response:
        return await response.json()

async def fetch_multiple_urls(urls):
    """Fetch multiple URLs concurrently"""
    async with aiohttp.ClientSession() as session:
        tasks = [fetch_url(session, url) for url in urls]
        results = await asyncio.gather(*tasks)
    return results

def lambda_handler(event, context):
    """
    Synchronous handler calling async code
    """
    urls = event.get('urls', [
        'https://api.example.com/user/1',
        'https://api.example.com/user/2',
        'https://api.example.com/user/3'
    ])
    
    # Run async function in Lambda (synchronous context)
    results = asyncio.run(fetch_multiple_urls(urls))
    
    return {
        'statusCode': 200,
        'body': json.dumps({
            'count': len(results),
            'results': results
        })
    }

Benefits:

  • Fetching 3 URLs sequentially: ~3 seconds
  • Fetching 3 URLs concurrently with async: ~1 second
  • Single Lambda instance handles all I/O concurrently without multiple threads/processes

Concurrent Lambda Invocations with asyncio.create_task​

Within a handler, spawn multiple tasks concurrently:

import asyncio
import json

async def process_item(item_id):
    """Simulate processing a single item"""
    print(f'Processing item {item_id}...')
    await asyncio.sleep(1)  # Simulate I/O
    return {'id': item_id, 'status': 'processed'}

async def process_batch(item_ids):
    """Process multiple items concurrently"""
    tasks = [process_item(item_id) for item_id in item_ids]
    results = await asyncio.gather(*tasks, return_exceptions=True)
    return results

def lambda_handler(event, context):
    item_ids = event.get('items', [1, 2, 3, 4, 5])
    
    # Run all tasks concurrently
    results = asyncio.run(process_batch(item_ids))
    
    successful = [r for r in results if not isinstance(r, Exception)]
    failed = [r for r in results if isinstance(r, Exception)]
    
    return {
        'statusCode': 200,
        'body': json.dumps({
            'successful': len(successful),
            'failed': len(failed),
            'results': successful
        })
    }

With 5 items taking 1 second each:

  • Sequential: 5 seconds total
  • Concurrent (asyncio): ~1 second total

Managing Lambda Reserved and Provisioned Concurrency​

Reserved Concurrency guarantees a minimum execution capacity and prevents one function from monopolizing account concurrency:

aws lambda put-function-concurrency \
  --function-name my-function \
  --reserved-concurrent-executions 100

# Now my-function can use up to 100 concurrent executions
# Other functions can use up to 900 (1000 account limit - 100 reserved)

Provisioned Concurrency pre-warms instances, eliminating cold starts for concurrent requests:

aws lambda put-provisioned-concurrent-executions \
  --function-name my-function \
  --provisioned-concurrent-executions 50

# AWS keeps 50 warm instances ready; eliminates cold starts for first 50 requests

Check current concurrency:

aws lambda get-function-concurrency --function-name my-function
# Output:
# {
#     "ReservedConcurrentExecutions": 100,
#     "UnreservedConcurrentExecutions": 900
# }

aws lambda get-provisioned-concurrency-config \
  --function-name my-function \
  --qualifier LIVE
# Output:
# {
#     "ProvisionedConcurrentExecutions": 50,
#     "AllocatedConcurrentExecutions": 50,
#     "AvailableConcurrentExecutions": 50,
#     "Status": "Ready"
# }

Batch Processing with Concurrent Lambda Invocations​

Scale across multiple Lambda instances for parallel processing:

import json
import boto3
import asyncio

lambda_client = boto3.client('lambda')

async def invoke_lambda(function_name, payload):
    """Asynchronously invoke a Lambda function"""
    loop = asyncio.get_event_loop()
    
    # Run blocking boto3 call in executor (thread pool)
    response = await loop.run_in_executor(
        None,
        lambda: lambda_client.invoke(
            FunctionName=function_name,
            InvocationType='RequestResponse',
            Payload=json.dumps(payload)
        )
    )
    
    return json.loads(response['Payload'].read())

async def process_batch_parallel(function_name, items):
    """Invoke Lambda concurrently for each item"""
    tasks = [
        invoke_lambda(function_name, {'item': item})
        for item in items
    ]
    results = await asyncio.gather(*tasks, return_exceptions=True)
    return results

def lambda_handler(event, context):
    items = event.get('items', list(range(100)))
    worker_function = event.get('worker_function', 'item-processor')
    
    # Invoke 100 separate Lambda instances concurrently
    results = asyncio.run(process_batch_parallel(worker_function, items))
    
    return {
        'statusCode': 200,
        'body': json.dumps({
            'total': len(items),
            'completed': sum(1 for r in results if not isinstance(r, Exception))
        })
    }

This orchestrator function scales to hundreds of worker Lambda instances without manual scaling.

Error Handling and Timeouts with Async​

Handle timeouts and exceptions in concurrent tasks:

import asyncio
import json

async def risky_operation(op_id):
    """Operation that may fail or timeout"""
    try:
        # Simulate work with timeout
        await asyncio.sleep(2)  # Takes 2 seconds
        if op_id % 3 == 0:
            raise ValueError(f'Operation {op_id} failed')
        return {'id': op_id, 'result': 'success'}
    except Exception as e:
        return {'id': op_id, 'error': str(e)}

async def process_with_timeout(op_ids, timeout=5):
    """Process operations with individual timeout"""
    tasks = [risky_operation(op_id) for op_id in op_ids]
    
    try:
        # Set overall timeout for all tasks
        results = await asyncio.wait_for(
            asyncio.gather(*tasks, return_exceptions=True),
            timeout=timeout
        )
        return results
    except asyncio.TimeoutError:
        print(f'Timeout exceeded ({timeout}s)')
        # Cancel remaining tasks
        for task in tasks:
            task.cancel()
        return []

def lambda_handler(event, context):
    op_ids = event.get('operations', [1, 2, 3, 4, 5])
    timeout = event.get('timeout', 10)
    
    results = asyncio.run(process_with_timeout(op_ids, timeout))
    
    successful = [r for r in results if 'error' not in r]
    failed = [r for r in results if 'error' in r]
    
    return {
        'statusCode': 200,
        'body': json.dumps({
            'successful': len(successful),
            'failed': len(failed)
        })
    }

Use asyncio.gather(..., return_exceptions=True) to continue processing even if some tasks fail.

Comparison: Lambda Concurrency Patterns​

PatternConcurrency TypeUse CaseComplexity
Async/awaitSingle instance, internal I/OAPI aggregation, multi-URL fetchesLow
Lambda invokeMultiple instancesBatch processing, fan-outMedium
SQS + LambdaAutomatic scalingDecoupled queues, backpressureMedium
Step FunctionsOrchestrated workflowComplex multi-step processesHigh

For simple parallel tasks, async/await is fastest and cheapest. For distributed workflows, use Lambda invoke or Step Functions.

Key Takeaways​

  • Async/await enables a single Lambda invocation to handle thousands of concurrent I/O tasks using event loops.
  • asyncio.run() is the entry point for running async code in a synchronous Lambda handler.
  • Lambda automatically scales to thousands of concurrent invocations across separate instances; combine with async for hybrid concurrency.
  • Reserved concurrency guarantees capacity; provisioned concurrency eliminates cold starts.
  • Use asyncio.gather(..., return_exceptions=True) to handle failures in concurrent tasks robustly.

Frequently Asked Questions​

Should I use threads or async in Lambda?​

Use async. Threads have high overhead (~2 MB stack each) and are limited by Python's GIL (Global Interpreter Lock). Async tasks are lightweight and non-blocking, allowing a single Lambda instance to handle thousands of concurrent I/O operations.

Can I use asyncio with all Python libraries?​

Only libraries designed for async (e.g., aiohttp, aioboto3, asyncpg) are safe. Libraries like requests and boto3 are synchronous and block the event loop. Wrap them with run_in_executor() if necessary, but prefer async alternatives.

What happens if my async tasks exceed the Lambda timeout?​

Lambda terminates the function and returns a timeout error. Remaining tasks are cancelled. Use asyncio.wait_for(timeout=...) to enforce per-task timeouts and gracefully handle overruns.

How do I debug async code in Lambda?​

Log at key points:

print(f'Starting task {task_id}')
try:
    result = await risky_operation()
    print(f'Task {task_id} completed: {result}')
except Exception as e:
    print(f'Task {task_id} failed: {str(e)}')

CloudWatch Logs capture all print statements in chronological order, showing concurrent task progress.

Is there a limit to concurrent async tasks?​

No hard limit within a single Lambda invocation, but practical limits depend on memory and timeout. A single 1024 MB, 15-minute function could theoretically handle millions of I/O tasks, but consider error handling and monitoring overhead.

Further Reading​