How to Handle Redirects When Scraping (3xx Explained)

Quick answer: To handle redirects when scraping, let your HTTP client follow 3xx responses automatically but cap the number of hops (five to ten) to avoid loops, resolve relative Location headers against the current URL, and carry cookies forward so session-based redirects work. Most clients follow redirects by default, but you should decide deliberately whether to follow them, because a redirect can leak you off-site or into an infinite loop.

Redirects are one of those things that work silently until they do not. Then you are staring at a scraper that returns a login page, an empty body, or a "too many redirects" error, and it is not obvious why. This post explains how 3xx status codes actually work and how to handle each case in a scraper.

The 3xx family, code by code

A redirect is a response in the 3xx range with a Location header telling the client where to go next. The differences between the codes matter more than most people think, especially for non-GET requests.

Code	Name	Method preserved?	Cacheable?	Typical use
301	Moved Permanently	Often changed to GET	Yes	URL moved for good
302	Found	Often changed to GET	No	Temporary redirect
303	See Other	Always switch to GET	No	After a POST, go GET a result
307	Temporary Redirect	Yes, method kept	No	Temporary, keep POST as POST
308	Permanent Redirect	Yes, method kept	Yes	Permanent, keep POST as POST

The historically messy part is method handling. For 301 and 302, many clients and browsers rewrite a POST into a GET when following the redirect, which is technically non-compliant but became a de facto standard. 303 says do this on purpose. 307 and 308 were added precisely to remove the ambiguity: they guarantee the method and body are preserved. When you scrape with GET requests this rarely bites you, but the moment you POST a form and get redirected, the code matters.

Following vs not following

Most HTTP clients follow redirects by default. That is usually what you want for scraping, since a 301 to the canonical URL is exactly the page you were after. But there are cases where you want to see the redirect instead of chasing it:

• Detecting canonical URLs. If you want to record that /old-path now points to /new-path, you need to see the 301 and its Location, not silently land on the destination.
• Avoiding off-site jumps. A redirect can send you to a completely different domain. If you only want to stay on the target site, inspect the Location before following.
• Login walls. A 302 to /login is a strong signal you were blocked or logged out, and following it just gives you a login form to parse by mistake.

The practical answer is to follow redirects but keep the final URL and the chain so you can reason about what happened.

The three things that break scrapers

Redirect loops

Site A redirects to B, B redirects back to A, and your client spins until it errors or hangs. Every well-behaved client caps the number of redirects it will follow (browsers use around twenty; libraries often default lower). Always keep a cap. If you build the following logic yourself, count hops and stop.

Relative vs absolute Location

The Location header is allowed to be a relative URL like /new-path or even ../thing. You must resolve it against the URL you just requested, not against the site root and not as-is. Getting this wrong sends requests to malformed URLs. Good HTTP clients resolve relative locations for you, which is a strong reason to lean on the client rather than parsing Location by hand.

Cookies across redirects

Session-based flows often set a cookie on the first response and expect it back on the redirect target. If your client does not carry cookies forward across the hop, the destination sees an unauthenticated request and bounces you to a login or a 403. Use a client with a cookie store, or a session object, so cookies persist across the chain. If you are seeing a redirect to a login page you did not expect, dropped cookies are the usual culprit.

Handling redirects in code

Here is Python with requests, which follows redirects by default and exposes the full chain.

import requests

resp = requests.get("https://example.com/old-path", allow_redirects=True)

print("final url:", resp.url)
print("status:", resp.status_code)

for hop in resp.history:
    print(hop.status_code, "->", hop.headers.get("Location"))

To inspect a redirect without following it, turn following off.

resp = requests.get("https://example.com/old-path", allow_redirects=False)
if resp.is_redirect:
    print("would go to:", resp.headers["Location"])

Use a Session when cookies must survive the chain.

session = requests.Session()  # persists cookies across redirects
resp = session.get("https://example.com/login-required")

In Node with fetch, redirects are followed automatically. Set redirect: "manual" to inspect instead.

const res = await fetch("https://example.com/old-path", { redirect: "manual" });
if (res.status >= 300 && res.status < 400) {
  console.log("redirect to:", res.headers.get("location"));
}

For more on what each status code means for a scraper beyond the 3xx range, see HTTP status codes for web scraping. A stubborn 403 after a redirect is often a blocking signal rather than a routing one, which we cover in HTTP 403 forbidden when scraping.

How a managed API handles redirects for you

When you fetch through the link.sc API, redirects are followed on the server side and you get the final rendered content, with cookies and relative locations resolved along the way. You do not write hop counting or Location parsing.

curl -X POST https://api.link.sc/v1/fetch \
  -H "x-api-key: lsc_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com/old-path", "format": "markdown"}'

The response is the destination page as clean markdown:

{ "content": "# The Real Page\n\nContent after all redirects were followed..." }

If you need to send your own cookies through the chain (for a session you control legitimately), the fetch body accepts cookies and headers, so the request that lands on the final URL carries what it needs. This is the same reason the managed approach is handy for JavaScript-heavy sites: the hard parts happen before the content reaches you.

A quick checklist

• Follow redirects, but cap the hop count to avoid loops.
• Keep the final URL and the redirect chain for debugging.
• Resolve relative Location headers against the current request URL.
• Persist cookies across hops with a session or cookie store.
• Treat an unexpected redirect to /login as a blocking signal, not a routing detail.
• Mind method preservation: 307 and 308 keep your method, 301, 302, and 303 may switch it to GET.

Redirects are simple once you know the rules, and dangerous when you assume the defaults are always right. Decide deliberately whether to follow, keep a cap, and carry cookies, and most 3xx surprises disappear.

---

Want redirects, cookies, and rendering handled for you? Get a free link.sc key and fetch any URL as clean markdown.