Async HTTP Requests in Python: aiohttp Guide
The most common use case for asyncio is handling I/O-bound tasks, and the most common I/O task in modern software is making network requests. However, Python's popular requests library is synchronous—if used in an async function, it blocks the entire event loop, defeating the purpose of asyncio. To perform HTTP requests asynchronously, you need a library purpose-built for non-blocking I/O. Aiohttp is the standard choice: it provides an asynchronous HTTP client and server, enabling you to make hundreds of concurrent network requests without blocking.
Key Takeaways
- Aiohttp is the async equivalent of the
requestslibrary; use it in async code instead ofrequests - ClientSession is the main object for making HTTP requests; create one and reuse it for all requests in your application
- Nested
async withblocks manage both the session and individual responses safely, ensuring proper cleanup - Asyncio.gather() combined with aiohttp enables concurrent requests: fetch 100 URLs in the time it takes to fetch one
- Error handling (
aiohttp.ClientError,raise_for_status()) is essential for robust async HTTP code
Installation and Setup
Aiohttp is a third-party library, so you'll need to install it with pip in your active virtual environment:
pip install aiohttp
Verify the installation:
python -c "import aiohttp; print(aiohttp.__version__)"
How Do You Make a Single Async HTTP Request?
Making an HTTP request with aiohttp involves two key components: creating a ClientSession and making the request within async with blocks. The session object manages connection pooling and reuse; the response object must be used within its own async with block.
Here's the pattern:
import aiohttp
import asyncio
async def fetch_single_url():
"""Fetch a single URL asynchronously."""
url = "https://jsonplaceholder.typicode.com/posts/1"
print("Starting request...")
# 1. Create a session (best practice: create once, reuse for all requests)
async with aiohttp.ClientSession() as session:
# 2. Make the GET request (await the coroutine)
async with session.get(url) as response:
# 3. Access response data
print(f"Status Code: {response.status}")
# .json() is a coroutine; must be awaited
data = await response.json()
print(f"Title: {data.get('title', 'N/A')}")
# Alternatively, get raw text:
# text_data = await response.text()
asyncio.run(fetch_single_url())
Output:
Starting request...
Status Code: 200
Title: sunt aut facere aut rerum necessitatibus sinc...
Key patterns:
- Outer
async with aiohttp.ClientSession()— manages the session for connection reuse - Inner
async with session.get(url)— manages the response object await response.json()— asynchronously reads and parses the JSON body- Both blocks are necessary for proper resource cleanup
How Do You Make Concurrent HTTP Requests?
The real power of aiohttp + asyncio is making many requests concurrently. asyncio.gather() runs multiple coroutines in parallel, so you fetch hundreds of URLs in the time of the slowest single request:
import aiohttp
import asyncio
import time
async def fetch_post_title(session, post_id: int):
"""Fetch a single post title asynchronously."""
url = f"https://jsonplaceholder.typicode.com/posts/{post_id}"
try:
async with session.get(url) as response:
# Check for HTTP errors (4xx, 5xx)
response.raise_for_status()
data = await response.json()
return data.get('title', 'N/A')
except aiohttp.ClientError as e:
print(f"Error fetching post {post_id}: {e}")
return None
async def main():
start_time = time.time()
# Create ONE session for all requests
async with aiohttp.ClientSession() as session:
# Create a list of coroutines (not awaiting yet)
tasks = [fetch_post_title(session, i) for i in range(1, 11)]
# Run all tasks concurrently
titles = await asyncio.gather(*tasks)
elapsed = time.time() - start_time
print(f"Fetched {len(titles)} posts in {elapsed:.2f} seconds")
print("\n--- First 3 Titles ---")
for i, title in enumerate(titles[:3], 1):
if title:
print(f"{i}. {title[:50]}...")
asyncio.run(main())
Output:
Fetched 10 posts in 0.85 seconds
--- First 3 Titles ---
1. sunt aut facere aut rerum necessitatibus sinc...
2. qui est esse...
3. ea molestias quasi exercitationem repellat qui...
Why this is fast: If these were synchronous requests, fetching 10 URLs would take ~10 × (time per request). With asyncio.gather(), all requests start immediately, and they complete in parallel. The total time is roughly equal to one request.
Accessing Different Response Types
Aiohttp responses can return data in multiple formats. Here's how to handle each:
import aiohttp
import asyncio
async def fetch_and_parse():
url = "https://jsonplaceholder.typicode.com/posts/1"
async with aiohttp.ClientSession() as session:
async with session.get(url) as response:
# JSON response
json_data = await response.json()
print(f"JSON: {json_data.get('title')}")
# Plain text response
# text_data = await response.text()
# Raw bytes
# bytes_data = await response.read()
# Status code and headers
print(f"Status: {response.status}")
print(f"Content-Type: {response.headers.get('Content-Type')}")
asyncio.run(fetch_and_parse())
Error Handling in Async HTTP Code
Network requests often fail. Proper error handling is essential:
import aiohttp
import asyncio
async def fetch_with_error_handling(session, url):
"""Fetch a URL with comprehensive error handling."""
try:
async with session.get(url, timeout=aiohttp.ClientTimeout(total=5)) as response:
# Raise an exception for bad status codes (4xx, 5xx)
response.raise_for_status()
return await response.json()
except asyncio.TimeoutError:
print(f"Timeout: {url} took too long")
return None
except aiohttp.ClientError as e:
# Covers connection errors, redirects, bad status codes (if not raised)
print(f"Client error fetching {url}: {e}")
return None
except asyncio.CancelledError:
print(f"Request cancelled: {url}")
raise # Re-raise cancellation
async def main():
async with aiohttp.ClientSession() as session:
tasks = [
fetch_with_error_handling(session, "https://jsonplaceholder.typicode.com/posts/1"),
fetch_with_error_handling(session, "https://httpstat.us/500"), # Will fail
fetch_with_error_handling(session, "https://httpstat.us/404"), # Will fail
]
results = await asyncio.gather(*tasks, return_exceptions=False)
print(f"Results: {results}")
asyncio.run(main())
Real-World Example: Web Scraping API Data
Here's a practical scenario: fetch data from 50 public APIs concurrently:
import aiohttp
import asyncio
# Sample list of public JSON APIs
APIS = [
"https://jsonplaceholder.typicode.com/users/1",
"https://jsonplaceholder.typicode.com/posts/1",
"https://api.github.com/zen", # Returns a single line
]
async def fetch_api(session, url):
"""Fetch a single API endpoint."""
try:
async with session.get(url, timeout=aiohttp.ClientTimeout(total=10)) as response:
response.raise_for_status()
# Some endpoints return JSON, some return text
try:
data = await response.json()
return {"url": url, "status": response.status, "data": data}
except:
text = await response.text()
return {"url": url, "status": response.status, "data": text[:100]}
except Exception as e:
return {"url": url, "status": "error", "data": str(e)}
async def fetch_multiple_apis(urls):
"""Fetch multiple APIs concurrently."""
async with aiohttp.ClientSession() as session:
tasks = [fetch_api(session, url) for url in urls]
results = await asyncio.gather(*tasks)
return results
# Run the example
results = asyncio.run(fetch_multiple_apis(APIS))
for result in results:
print(f"URL: {result['url']}")
print(f"Status: {result['status']}")
print(f"Data: {str(result['data'])[:80]}...")
print()
Frequently Asked Questions
Why must you use aiohttp instead of requests in async code?
The requests library is synchronous and blocks the entire thread during I/O. Using it in an async function defeats the purpose of asyncio—other tasks cannot run while one request is pending. Aiohttp is designed for non-blocking I/O, allowing thousands of requests to run concurrently without blocking.
How many concurrent requests can aiohttp handle?
Theoretically, thousands. Practically, limits depend on system resources (file descriptors, memory), the remote server (rate limiting, connection limits), and your network bandwidth. Most servers limit concurrent connections per client. For heavy workloads, use a TCPConnector with connection limits to be a good citizen: aiohttp.ClientSession(connector=aiohttp.TCPConnector(limit=10)).
What is the difference between response.json() and await response.json()?
response.json() returns a coroutine; you must await it to actually read and parse the response body. This is why you write data = await response.json(). Forgetting await gives you a coroutine object, not the data. Similarly, response.text() and response.read() require await.
Can you reuse a ClientSession for multiple requests?
Yes—in fact, you should. Create one session and reuse it for all requests in your program. Sessions manage connection pooling, which greatly improves performance by reusing TCP connections. Create a new session only when necessary (e.g., switching authentication credentials).
How do you add custom headers or authentication to requests?
Pass headers as a parameter:
headers = {"Authorization": "Bearer YOUR_TOKEN", "User-Agent": "MyBot/1.0"}
async with session.get(url, headers=headers) as response:
data = await response.json()
For OAuth or Basic Auth, aiohttp has built-in support; see the official docs.
Conclusion
Aiohttp is the standard tool for asynchronous HTTP in Python. By combining it with asyncio.gather(), you unlock the ability to fetch hundreds or thousands of URLs concurrently, transforming slow sequential operations into lightning-fast parallel ones. This is essential for modern applications: web scrapers, API aggregators, bulk data processing, and real-time data pipelines.
The pattern is straightforward: create one session, build a list of coroutines, gather them concurrently, and process results. Master this, and you'll dramatically improve the performance of any I/O-bound Python application.