Skip to main content

HTTP Requests and Beautiful Soup for Web Parsing

HTTP (HyperText Transfer Protocol) is the foundation of web communication. Every time a scraper fetches a page, it sends an HTTP request with headers that identify itself and receive a response containing status codes, headers, and the page body. BeautifulSoup then parses that HTML into a tree structure you can query. Mastering both HTTP mechanics and BeautifulSoup's API is critical to building scrapers that handle real-world variations, errors, and complex nested HTML structures. This article dives deep into both, with practical patterns for every common scraping scenario.

I spent months struggling with "mysterious" 403 errors and missing data before I learned that many servers inspect the User-Agent header and block requests that look automated. Once I understood HTTP headers and BeautifulSoup's traversal methods, my scraper reliability jumped from 60% to 98%. These fundamentals matter.

Understanding HTTP Headers and Requests​

When you send an HTTP request, you include headers that communicate metadata about the request. The server uses these to decide whether to serve you, redirect you, or block you. The requests library lets you control every aspect of the request, including headers, cookies, and timeouts.

Here is how to craft a request that identifies itself properly and handles responses:

import requests
from datetime import datetime

# Define headers that make your scraper look like a real browser
headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.5",
    "Referer": "https://www.google.com/",
    "DNT": "1",
    "Connection": "keep-alive",
    "Upgrade-Insecure-Requests": "1"
}

url = "https://example.com/products"

try:
    # Send GET request with custom headers and 15-second timeout
    response = requests.get(
        url,
        headers=headers,
        timeout=15,
        allow_redirects=True
    )
    
    # Check status code
    if response.status_code == 200:
        print(f"Success! Page size: {len(response.content)} bytes")
    elif response.status_code == 429:
        print("Rate limited. Wait before retrying.")
    elif response.status_code == 403:
        print("Forbidden. Server rejected request.")
    else:
        print(f"HTTP {response.status_code}: {response.reason}")
    
    # Inspect response headers
    print(f"Content-Type: {response.headers.get('Content-Type')}")
    print(f"Server: {response.headers.get('Server')}")
    
except requests.exceptions.Timeout:
    print("Request timed out after 15 seconds.")
except requests.exceptions.ConnectionError:
    print("Network error. Server unreachable.")
except requests.exceptions.RequestException as e:
    print(f"Request failed: {e}")

Key points:

  • User-Agent: Identify yourself as a browser (not a scraper) to avoid automatic blocks.
  • Timeout: Prevent hanging requests; 10-15 seconds is reasonable.
  • Status codes: 200 = OK, 301/302 = redirect (requests follows automatically by default), 403 = forbidden, 429 = rate limited, 500 = server error.
  • Exception handling: Network errors are normal; catch them to build resilient scrapers.

Parsing HTML with BeautifulSoup Selectors​

BeautifulSoup supports two primary ways to find elements: CSS selectors (fast, intuitive) and Tag traversal (slower, more flexible). Most scrapers use selectors. Here is a comprehensive example:

from bs4 import BeautifulSoup
import requests

html = """
<html>
  <body>
    <div class="product-list">
      <article class="product" data-id="101">
        <h2 class="product-title">Laptop Pro</h2>
        <span class="price">$999</span>
        <p class="description">High-performance laptop</p>
        <a href="/product/101" class="view-link">View Details</a>
      </article>
      <article class="product" data-id="102">
        <h2 class="product-title">Phone X</h2>
        <span class="price">$799</span>
        <p class="description">Latest smartphone</p>
        <a href="/product/102" class="view-link">View Details</a>
      </article>
    </div>
  </body>
</html>
"""

soup = BeautifulSoup(html, "html.parser")

# Select all products
products = soup.select("article.product")
print(f"Found {len(products)} products\n")

for product in products:
    # CSS selectors for finding elements within each product
    title = product.select_one("h2.product-title").get_text(strip=True)
    price = product.select_one("span.price").get_text(strip=True)
    description = product.select_one("p.description").get_text(strip=True)
    
    # Extract attributes
    product_id = product.get("data-id")
    link = product.select_one("a.view-link").get("href")
    
    print(f"ID: {product_id}")
    print(f"Title: {title}")
    print(f"Price: {price}")
    print(f"Description: {description}")
    print(f"Link: {link}\n")

Common CSS selector patterns:

  • .classname = element with class
  • #id = element with ID
  • tag.class = tag with specific class
  • parent > child = direct child
  • ancestor descendant = any descendant
  • [attribute=value] = element with attribute value
  • :nth-child(n) = nth child of parent

Real-world HTML is messy: deeply nested, with inconsistent classes, and missing elements. You must handle these gracefully:

from bs4 import BeautifulSoup

html = """
<div class="results">
  <div class="result-item">
    <div class="header">
      <h3>Article Title</h3>
      <span class="author">John Doe</span>
    </div>
    <div class="body">
      <p>Article content here...</p>
    </div>
    <div class="metadata">
      <span class="date">2026-06-02</span>
    </div>
  </div>
  <div class="result-item">
    <!-- Missing author span in second item -->
    <div class="header">
      <h3>Another Article</h3>
    </div>
    <div class="body">
      <p>More content...</p>
    </div>
  </div>
</div>
"""

soup = BeautifulSoup(html, "html.parser")

for item in soup.select("div.result-item"):
    # Use .select_one() and check for None to handle missing elements
    title_elem = item.select_one("h3")
    title = title_elem.get_text(strip=True) if title_elem else "Unknown"
    
    author_elem = item.select_one("span.author")
    author = author_elem.get_text(strip=True) if author_elem else "Unknown"
    
    content_elem = item.select_one("div.body p")
    content = content_elem.get_text(strip=True) if content_elem else ""
    
    date_elem = item.select_one("span.date")
    date = date_elem.get_text(strip=True) if date_elem else None
    
    print(f"Title: {title}")
    print(f"Author: {author}")
    print(f"Content: {content[:50]}...")
    print(f"Date: {date}\n")

Always use .select_one() with a None check rather than assuming an element exists. This prevents your scraper from crashing on minor HTML variations.

Extracting Text and Attributes Cleanly​

Text extraction has quirks: whitespace, newlines, and nested tags can pollute your data. Here are proven techniques:

from bs4 import BeautifulSoup

html = """
<div class="product">
  <h2>  Laptop  \n  Pro  </h2>
  <p>Price: <strong>$999</strong> (on sale)</p>
  <img src="/img/laptop.jpg" alt="Product image" />
  <a href="https://example.com/product/101" data-category="electronics">Buy Now</a>
</div>
"""

soup = BeautifulSoup(html, "html.parser")
product = soup.select_one("div.product")

# Get text with strip=True to remove extra whitespace
title = product.select_one("h2").get_text(strip=True)
print(f"Title: {title}")  # Output: "Laptop Pro"

# Extract text between tags (ignore nested tags)
price_text = product.select_one("p").get_text(strip=True)
print(f"Price text: {price_text}")  # Output: "Price: $999 (on sale)"

# Extract only the currency value using string methods
price_value = price_text.split("$")[1].split(" ")[0]
print(f"Price value: {price_value}")  # Output: "999"

# Get image source
img = product.select_one("img")
img_src = img.get("src") if img else None
img_alt = img.get("alt") if img else None
print(f"Image: {img_src} (alt: {img_alt})")

# Get link and multiple attributes
link_elem = product.select_one("a")
href = link_elem.get("href")
category = link_elem.get("data-category")
print(f"Link: {href}, Category: {category}")

Key patterns:

  • .get_text(strip=True) removes leading/trailing whitespace and collapses newlines.
  • .get("attr") reads any HTML attribute; returns None if missing.
  • Chain .get_text() with string methods (.split(), .replace()) to extract structured data.

Request Session Management for Efficiency​

When making many requests, reuse a Session object to maintain cookies and connection pooling:

import requests
from bs4 import BeautifulSoup

# Create a session that persists cookies and headers
session = requests.Session()
session.headers.update({
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
})

# Make multiple requests with the same session
urls = [
    "https://example.com/page/1",
    "https://example.com/page/2",
    "https://example.com/page/3"
]

for url in urls:
    try:
        response = session.get(url, timeout=10)
        response.raise_for_status()
        
        soup = BeautifulSoup(response.text, "html.parser")
        items = soup.select("div.item")
        print(f"{url}: Found {len(items)} items")
        
    except requests.exceptions.RequestException as e:
        print(f"{url}: Error - {e}")

# Session closes automatically at the end, but you can force it
session.close()

Sessions are faster because they reuse TCP connections and cookies (useful if the site tracks your login across pages).

Key Takeaways​

  • HTTP headers like User-Agent, Referer, and Accept tell servers about your request and whether you are a browser or scraper.
  • Always handle HTTP status codes and exceptions (timeout, connection error) to build robust scrapers.
  • CSS selectors are the fastest way to find elements; always check for None before accessing nested elements.
  • Use .get_text(strip=True) to clean text and .get("attribute") to read HTML attributes.
  • Session objects reuse connections and cookies, improving speed for multi-page scraping.

Frequently Asked Questions​

What is the difference between .select() and .select_one()?​

.select() returns a list of all matching elements (empty list if none found). .select_one() returns the first match or None if no match. Use .select_one() when you expect only one element and want a None check to prevent errors.

Why am I getting 403 Forbidden errors?​

Many sites block requests without a realistic User-Agent header or reject requests that come too quickly. Add headers that mimic a browser and space out your requests with delays. Some sites also check Referer and Cookie headers.

How do I find the right CSS selector for an element?​

Open the page in your browser, right-click the element, and select "Inspect Element." Look at the HTML structure and identify unique classes or IDs. Test the selector in a Python REPL before using it in your scraper. Use browser DevTools to verify your selectors work.

Can I parse XML or JSON with BeautifulSoup?​

BeautifulSoup primarily parses HTML, but it works with XML too if you specify "xml" as the parser. For JSON, use Python's built-in json module: data = json.loads(response.text). Many modern sites return JSON APIs instead of HTML.

How do I handle encoding issues in text?​

BeautifulSoup and requests handle encoding automatically in most cases. If you encounter mojibake (garbled text), explicitly specify encoding: response.encoding = "utf-8" before accessing response.text. Check the server's Content-Type header for the declared encoding.

Further Reading​