← All posts

HTTP Status Codes for Web Scraping: The Ones That Matter

ENGINEERING · JUL 18, 2026 · 6 MIN READ

Quick answer: For web scraping, six groups of HTTP status codes cover almost everything you hit: 200 means success and parse the body; 3xx means follow the redirect; 403 means you were blocked, so fix your fingerprint; 404 means the page is gone, so drop it; 429 means slow down and back off; and 5xx means the server erred, so retry with a delay. Knowing which is which turns a scraper that crashes into one that adapts.

Every response your scraper gets carries a status code, and treating them all the same is the difference between a robust crawler and a brittle script that dies on the first hiccup. This is the roundup I wish I had when I started: what each code means specifically for a scraper, and the correct reaction to each.

The Quick Reference Table

Code Meaning Scraper reaction
200 OK Success, body is valid Parse it
301 / 308 Permanent redirect Follow, update the stored URL
302 / 307 Temporary redirect Follow, keep the original URL
304 Not Modified Cached copy still valid Use your cached version
403 Forbidden Blocked or not authorized Fix headers/fingerprint or back off
404 Not Found Page does not exist Drop the URL, do not retry
410 Gone Permanently removed Drop it, remove from the frontier
429 Too Many Requests Rate limited Back off, respect Retry-After
500 / 502 / 503 / 504 Server error Retry with exponential backoff

The rest of this post is what sits behind each row.

200: Success, But Verify the Body

A 200 OK means the request succeeded and the response body is the content you asked for. This is what you want, but do not trust the code alone. Some sites return 200 with a CAPTCHA page, a "please enable JavaScript" shell, or an empty container that JavaScript fills in later. So a 200 should trigger a content check: does the body contain the elements you expected to parse? If not, you have a soft block wearing a success code. I unpack the details in the HTTP 200 status code explained.

3xx: Redirects You Should Usually Follow

Redirects are routine. A 301 or 308 is permanent (the resource moved for good), so follow it and update the URL you have stored, or you will keep taking the extra hop forever. A 302 or 307 is temporary, so follow it but keep the original URL as canonical.

Most HTTP clients follow redirects automatically, which is convenient until it hides a problem: a site can redirect a suspected bot to a login or a block page and return a happy 200 at the end of the chain. Log the final URL after redirects and check it is where you meant to go.

import requests

resp = requests.get("https://example.com/page", allow_redirects=True)
if resp.history:
    print("Redirect chain:", [r.status_code for r in resp.history])
    print("Landed on:", resp.url)

304 Not Modified is the redirect family's useful cousin: send an If-Modified-Since or If-None-Match header, and a 304 tells you your cached copy is still current, so you skip the download entirely. That is free bandwidth and speed on recrawls.

403: You Have Been Blocked

A 403 Forbidden in scraping almost never means "this data is private." It usually means the site's bot management decided you are not a real browser. The common causes:

  • A default client user agent like python-requests/2.31.0.
  • Missing or wrongly ordered headers (Accept, Accept-Language, client hints).
  • A TLS fingerprint that says "scripting library" while your user agent says "Chrome."
  • A datacenter IP with poor reputation.

The fix is consistency, not brute force. Set a realistic browser identity across every layer, not just the user agent. Two posts go deep here: rotate user agents and headers correctly and TLS fingerprinting and bot detection. Retrying a 403 with the exact same fingerprint just wastes requests.

404 and 410: The Page Is Gone

A 404 Not Found means the URL does not resolve to a resource. Do not retry it; a 404 will not heal. Log it, drop it from your queue, and move on. If you are seeing many 404s, your URL discovery is probably generating bad links (guessed IDs, stale sitemap entries).

A 410 Gone is a stronger 404: the resource existed and was deliberately removed. Treat it as permanent and prune the URL from your crawl frontier so you never queue it again.

429: Slow Down

A 429 Too Many Requests is the server telling you, explicitly, that you are going too fast. This is the one code you should be almost grateful for, because it is honest feedback instead of a silent block. React by backing off, and check for a Retry-After header that tells you exactly how long to wait:

import time
import requests

def get_with_backoff(url, headers, max_retries=5):
    delay = 1
    for attempt in range(max_retries):
        resp = requests.get(url, headers=headers)
        if resp.status_code == 429:
            wait = int(resp.headers.get("Retry-After", delay))
            time.sleep(wait)
            delay *= 2  # exponential backoff
            continue
        return resp
    return resp

The durable fix is to not hit 429 in the first place: cap your concurrency, add delays between requests, and spread load. A crawler that respects rate limits gets blocked far less often. The full playbook is in the HTTP 429 error explained and how to fix it.

5xx: The Server's Problem, So Retry

The 5xx family means the server failed, not you. 500 is a generic internal error, 502 a bad gateway, 503 service unavailable (often overload or maintenance), and 504 a gateway timeout. These are frequently transient, so the right reaction is retry with exponential backoff and a cap, exactly like the 429 code above.

One special case worth knowing: Cloudflare's 520 is a non-standard code meaning the origin server returned something Cloudflare could not understand. It often shows up under heavy or unusual request patterns and can indicate the origin is struggling or that your requests look abnormal. I wrote it up separately in Cloudflare error 520 explained. For the standard 5xx codes, back off and retry; if a specific URL returns 5xx every time for hours, treat it like a soft 404 and shelve it.

A Note on Ethics

Status codes are also consent signals. A 429 is the site asking you to slow down, and a 403 after a polite request may be the site asking you not to scrape at all. Respect robots.txt, honor Retry-After, keep your request rate reasonable, and do not treat repeated blocks as a puzzle to defeat at any cost. Reacting correctly to these codes is not just robustness; it is good citizenship on someone else's infrastructure.

Let link.sc Absorb the Status-Code Churn

Handling every code correctly (redirect chains, backoff on 429 and 5xx, fingerprint fixes for 403, soft-block detection on 200) is a lot of plumbing to build and maintain. link.sc does it inside the fetch layer, retrying, rotating, and rendering as needed, then returns either the clean content or a clear error:

curl https://link.sc/v1/fetch \
  -H "Authorization: Bearer lsc_..." \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com/page", "format": "markdown"}'

You get a parsed result instead of a status code to babysit. The free tier is 500 credits a month, enough to see how much retry logic you can delete.

The Bottom Line

You do not need to memorize the full HTTP spec to scrape well. You need to react correctly to six situations: parse on 200, follow on 3xx, fix your fingerprint on 403, drop on 404, back off on 429, and retry on 5xx. Build those reactions in once and your scraper stops crashing on the first surprise and starts handling the web the way it actually behaves.

Want fetches that already handle redirects, blocks, and rate limits for you? link.sc returns clean content, not status codes to babysit. Start free.