Skip to main content

Working with Async HTTP Requests: `aiohttp`

The most common use case for asyncio is to handle I/O-bound tasks, and the most common I/O task in modern software is making network requests. However, the popular requests library is synchronous. If you use it in an async function, it will block the entire event loop, defeating the purpose of asyncio.

To perform HTTP requests asynchronously, you need a library designed for it. The most popular and robust library for this is aiohttp. It provides an asynchronous HTTP client and server, allowing you to make many network requests concurrently without blocking.

📚 Prerequisites​

You should be comfortable with asyncio fundamentals, including async/await and running concurrent tasks with asyncio.gather().

🎯 Article Outline: What You'll Master​

In this article, you will learn:

  • ✅ Installing aiohttp: How to add the library to your environment.
  • ✅ The ClientSession: Understand the importance of using a session object for making requests.
  • ✅ Making a GET Request: How to perform a basic GET request to fetch data.
  • ✅ Processing the Response: How to get the status code, JSON data, and text from a response object.
  • ✅ Concurrent Requests: A practical example of using aiohttp with asyncio.gather() to fetch multiple URLs at once.

🧠 Section 1: Installation​

aiohttp is a third-party library, so you'll need to install it with pip. As always, it's highly recommended to do this in an active virtual environment.

pip install aiohttp

💻 Section 2: Making a Single Request​

Making a request with aiohttp involves two main components:

  1. aiohttp.ClientSession: This is the main entry point for all client-side requests. You should create a single session and reuse it for all requests in your application. The best way to manage it is with an async with block, which handles closing the session for you automatically.
  2. The Request: Inside the async with block, you can use methods on the session object, like session.get(), session.post(), etc. These methods are coroutines, so you must await them.

This also returns a response object that must be used within another async with block.

Let's fetch some data from a public test API.

import aiohttp
import asyncio

async def fetch_one():
    # A free public API for testing
    url = "https://jsonplaceholder.typicode.com/posts/1"

    print("Starting request...")
    # 1. Create a client session
    async with aiohttp.ClientSession() as session:
        # 2. Make the GET request
        async with session.get(url) as response:
            # 3. The response object has methods for accessing the data
            print(f"Status Code: {response.status}")

            # .json() is a coroutine, so it must be awaited
            data = await response.json()
            print(f"Response JSON: {data}")

            # .text() is also a coroutine
            # text_data = await response.text()

asyncio.run(fetch_one())

Output:

Starting request...
Status Code: 200
Response JSON: {'userId': 1, 'id': 1, 'title': 'sunt aut facere ...', 'body': 'quia et suscipit...'}

Notice the nested async with blocks. This is the standard, safe pattern for using aiohttp. The outer block manages the session, and the inner block manages the response from a single request.

🛠️ Section 3: Concurrent Requests with asyncio.gather()​

The real power of aiohttp is realized when you combine it with asyncio.gather() to perform many requests concurrently. This is perfect for web scraping or querying multiple API endpoints at once.

Let's write a script to fetch the titles of the first 5 posts from the JSONPlaceholder API.

import aiohttp
import asyncio
import time

async def fetch_post_title(session, post_id: int):
    """Fetches a single post and returns its title."""
    url = f"https://jsonplaceholder.typicode.com/posts/{post_id}"
    try:
        async with session.get(url) as response:
            # Raise an exception for bad status codes (4xx or 5xx)
            response.raise_for_status()
            data = await response.json()
            return data['title']
    except aiohttp.ClientError as e:
        print(f"Request for post {post_id} failed: {e}")
        return None

async def main():
    start_time = time.time()

    # Create a single session to be reused for all requests
    async with aiohttp.ClientSession() as session:
        # Create a list of coroutines to run
        tasks = [fetch_post_title(session, i) for i in range(1, 6)]

        # Run them all concurrently
        titles = await asyncio.gather(*tasks)

    end_time = time.time()
    print(f"Fetched {len(titles)} titles in {end_time - start_time:.2f} seconds.")
    print("\n--- Titles ---")
    for i, title in enumerate(titles, 1):
        if title:
            print(f"{i}. {title[:40]}...") # Print first 40 chars

asyncio.run(main())

How it works:

  1. We create a single ClientSession in our main function. This is much more efficient than creating a new one for each request.
  2. We pass this session object as an argument to our fetch_post_title coroutine.
  3. We create a list of tasks, where each task is a call to fetch_post_title with a different post_id.
  4. asyncio.gather(*tasks) runs all these requests concurrently. The total time taken will be close to the time of the single slowest request, not the sum of all of them.

✨ Conclusion & Key Takeaways​

aiohttp is the standard and essential tool for any asyncio application that needs to communicate over HTTP. By integrating it with asyncio.gather(), you can build incredibly fast and efficient network clients.

Let's summarize the key takeaways:

  • Use an async library for async code: Standard libraries like requests will block the event loop. Use aiohttp instead.
  • Use a single ClientSession: Create one session and reuse it for multiple requests for better performance.
  • Use async with: This pattern ensures that both the session and the response are closed properly.
  • Combine with gather for concurrency: The true power of aiohttp is unlocked when you run many requests concurrently with asyncio.gather().

➡️ Next Steps​

Making network requests is just one part of I/O. What about reading and writing files? While standard file I/O can be blocking, there are libraries to handle this asynchronously as well. In the next article, we'll explore "Asynchronous File I/O: aiofiles."

Happy fetching!