HTTP 407 Proxy Authentication Required: How to Fix It When Scraping

Quick answer: HTTP 407 means the proxy server between you and the website refused your request because you didn't authenticate with the proxy itself. The target site never saw your request. The fix is almost always one of three things: pass credentials in the correct proxy URL format (http://user:pass@host:port), URL-encode special characters in the password, or add your current egress IP to the provider's allowlist if you're using IP-based auth.

Every other 4xx code is a message from the website you're trying to reach. 407 is the only status code in the spec that comes from the middlebox instead. That distinction is the whole key to debugging it, so let's start there.

407 Is a 401 From the Proxy, Not the Site

The two codes are mirror images. A 401 comes from the origin server with a WWW-Authenticate header and expects an Authorization header back. A 407 comes from the proxy with a Proxy-Authenticate header and expects Proxy-Authorization back.

	401 Unauthorized	407 Proxy Authentication Required
Who sent it	The website	Your proxy provider
Challenge header	WWW-Authenticate	Proxy-Authenticate
Credentials header	Authorization	Proxy-Authorization
Did the site see your request?	Yes	No, it never left the proxy

This matters for scraping because a 407 tells you your anti-bot setup, headers, and target URL are all irrelevant to the failure. You could be fetching example.com and you'd still get the 407. Don't waste time rotating user agents or checking whether the site blocks scrapers. Fix the proxy handshake first.

The Two Ways Proxies Authenticate

Residential and datacenter proxy providers offer two auth modes, and half of all 407s I've debugged come from mixing them up.

Username and password. Credentials ride along with every request, embedded in the proxy URL. Works from anywhere, survives IP changes, and is the right choice for cloud functions and containers whose egress IPs you don't control.

IP allowlisting. You register your server's public IP in the provider's dashboard, and any connection from that IP is accepted with no credentials at all. Convenient on a fixed VPS, but it fails silently the moment your infrastructure moves: a new deploy region, a NAT gateway change, a laptop on different wifi. The provider sees an unknown IP, and you see a 407.

The failure mode of allowlisting is sneaky because nothing in your code changed. If a scraper that ran fine for weeks suddenly starts throwing 407s, check what your egress IP is right now (curl ifconfig.me from the same machine) and compare it against the dashboard. Cloud environments recycle IPs constantly, which is why most providers recommend user-pass auth for anything serverless.

Getting the Credential Format Right

The standard format packs everything into the proxy URL:

curl -x "http://username:[email protected]:8000" https://example.com

Or split out, which avoids quoting problems:

curl -x "http://gate.provider.com:8000" \
     --proxy-user "username:password" \
     https://example.com

In Python with requests:

import requests

proxies = {
    "http": "http://user:[email protected]:8000",
    "https": "http://user:[email protected]:8000",
}
r = requests.get("https://example.com", proxies=proxies)

Note that the https entry still uses an http:// proxy URL. That's normal: the client makes a plain-HTTP CONNECT to the proxy and tunnels TLS through it. Writing https:// there is a classic source of confused 407s and connection errors.

Three format gotchas cause most of the rest:

1. Special characters in the password. If your password contains @, :, /, or #, the URL parser splits in the wrong place and sends garbage credentials. URL-encode them: p@ss becomes p%40ss. Better yet, regenerate the password without special characters. Every proxy dashboard lets you.

2. Session parameters belong in the username. Residential providers overload the username field with routing flags, something like user-country-us-session-abc123. Typos here don't produce a helpful error, just a 407 or a misrouted request. Copy the exact string from the provider's docs.

3. Credentials on the CONNECT request. For HTTPS targets, authentication happens on the CONNECT call, and some HTTP clients don't attach Proxy-Authorization there by default or refuse to send it preemptively. Older Java clients and some Axios versions are known offenders. If curl works but your code doesn't, this is the likely culprit, and the fix is usually a client-specific flag or switching proxy libraries.

A Two-Minute Debugging Sequence

Run these in order and you'll find the cause of nearly any 407:

# 1. Does the proxy accept your credentials at all?
curl -v -x "http://user:[email protected]:8000" https://httpbin.org/ip

# 2. What IP does the provider see you as? (for allowlist mode)
curl ifconfig.me

# 3. Are the credentials still valid? Check the provider dashboard.

In the verbose output, look for the CONNECT line and the Proxy-Authorization header. If the header is missing, your client isn't sending credentials. If it's present and you still get 407, the credentials are wrong, the IP isn't allowlisted, or your plan has lapsed. A few providers return 407 instead of 402 when your bandwidth balance hits zero, which is a spec-abusing but common choice, so check the billing page before rewriting any code.

One more thing worth knowing: keep retry logic away from 407s. Unlike a 429, which resolves itself if you back off, a 407 is deterministic. The same request will fail the same way a thousand times, and hammering the gateway can get your account flagged by the provider's abuse systems.

Why Managed Fetch APIs Make 407 Disappear

Here's the structural observation: 407 only exists in your life because you're managing proxy credentials yourself. Every scraper that talks to a proxy gateway has to store secrets, encode them correctly, keep allowlists in sync with shifting egress IPs, and monitor bandwidth balances across providers. Each of those is a 407 waiting to happen, and none of them is the actual work of extracting data.

A managed fetch API collapses that whole layer. When you call link.sc, the proxy fleet, rotation, and authentication all live behind one API key:

curl -H "Authorization: Bearer YOUR_API_KEY" \
     "https://api.link.sc/fetch?url=https://example.com"

There's no proxy URL to format, no password to encode, and no allowlist to update when your deploy moves regions. The escalation from plain requests through residential IPs to full browser rendering happens server-side, so the entire class of proxy-auth failures is gone from your codebase. The developer guide covers the setup in about five minutes.

If you'd rather keep running your own proxies, that's a legitimate choice at high volume. Just treat 407 as the configuration smoke alarm it is: it never means the website blocked you, and it always means the fix is in your proxy setup, not your scraping logic.

---

Skip proxy credential management entirely: sign up for link.sc and fetch any page with a single API key.