← All posts

How Do I Scrape a Page That Needs a Specific Geographic Location?

GUIDES · JUL 18, 2026 · 6 MIN READ

Quick answer: First check whether the site encodes location in the URL or a cookie, because forcing ?country=de or an Accept-Language header is free and often enough. If the site geolocates by IP address, you need to make the request from an IP in the target region, either through a geo-located proxy or a fetch API that supports geotargeting. Always verify what you actually received, because sites mix signals and will happily serve you a hybrid page.

Scrape a product page from a US server and you get dollars. Your customer in Germany sees euros, different shipping, sometimes a different price entirely. If your dataset is supposed to reflect what users in a specific market see, requests from the wrong location produce data that is wrong in ways no amount of parsing can fix. Here's how location actually gets decided, and how to control it.

Why the Same URL Serves Different Pages

Sites vary content by location for mostly legitimate reasons:

  • Currency and price. Regional pricing is standard in e-commerce, SaaS, and travel. The same SKU can have genuinely different prices per market, not just converted ones.
  • Language. Many sites auto-translate or redirect based on inferred location.
  • Availability. Products, streaming catalogs, and services that only exist in some markets.
  • Legal requirements. Cookie banners in the EU, age gates, region-locked content.
  • Redirects. The big one for crawlers: request example.com from a French IP and get bounced to example.com/fr/ before you've parsed a byte.

The practical consequence: "the page" is not one document. When someone reports that your scraped price doesn't match what they see, location mismatch should be your first suspect, right next to caching.

How Sites Decide Where You Are

Four signals, roughly in order of how often they dominate:

Signal How it works Can you fake it without a proxy?
IP geolocation GeoIP lookup on your request IP No
URL structure /de/, ?country=uk, de.example.com Yes, trivially
Cookies A region cookie set on first visit or by a country picker Yes
Accept-Language header Browser locale preference Yes

Most sites combine them, typically IP for the first guess and a cookie to remember your choice. That combination is exactly why you must verify results: send a German Accept-Language from a US IP and some sites give you German text with USD prices, a page no real user ever sees.

Try the Cheap Levers First

Before paying for geo infrastructure, spend ten minutes checking whether the site exposes location in the URL or cookies. A surprising fraction do, because country pickers have to work somehow.

Click the site's own country selector and watch what changes: the path, a query param, or a cookie. Then replicate it:

import requests

session = requests.Session()

# Lever 1: URL structure
r = session.get("https://example.com/de-de/product/123", timeout=20)

# Lever 2: query params
r = session.get("https://example.com/product/123",
                params={"country": "DE", "currency": "EUR"}, timeout=20)

# Lever 3: locale headers and region cookie
r = session.get(
    "https://example.com/product/123",
    headers={"Accept-Language": "de-DE,de;q=0.9"},
    cookies={"region": "DE"},
    timeout=20,
)

When these work, they're strictly better than proxies: faster, free, and deterministic. When they don't, it's usually because the site trusts IP over everything else, which brings us to the expensive lever.

When Only the IP Will Do

If the site geolocates by IP and ignores your hints, the request has to originate from the target region. Two ways to get there.

Geo-located proxies. Residential or datacenter proxies with country selection. You route your HTTP client through them:

import requests

proxies = {
    "https": "http://user-country-de:[email protected]:8000",
}
r = requests.get("https://example.com/product/123",
                 proxies=proxies, timeout=30)

This works, but you now own proxy pool management: rotation, dead exits, providers whose "Germany" IPs occasionally geolocate to Austria, and the interaction between proxies and anti-bot systems. Datacenter proxy IPs in particular are widely flagged, so a geo-correct request can still come back blocked; that failure mode looks confusingly like a geo problem but isn't (here's how to tell what a site actually requires).

A fetch API with geotargeting. The alternative is to let an API pick the exit location and handle the proxy plumbing, retries, and rendering in one call:

curl "https://link.sc/v1/fetch?url=https://example.com/product/123&country=de" \
  -H "Authorization: Bearer lsc_your_key"

Same idea from Python, as part of a multi-market price check:

import requests

API = "https://link.sc/v1/fetch"
HEADERS = {"Authorization": "Bearer lsc_your_key"}

def fetch_from(url: str, country: str) -> str:
    r = requests.get(API, headers=HEADERS,
                     params={"url": url, "country": country, "format": "markdown"},
                     timeout=60)
    r.raise_for_status()
    return r.text

for market in ["us", "de", "jp"]:
    page = fetch_from("https://example.com/product/123", market)
    print(market, extract_price(page))

My general rule: cheap levers if the site offers them, an API with geotargeting when it doesn't, and self-managed proxy pools only when you have a specific reason to own that layer. Details on the fetch parameters are in the link.sc docs.

Verify What You Actually Got

This is the step everyone skips and then regrets. Geo setups fail quietly: the proxy exit isn't where the provider claims, the site half-respects your header, a CDN serves you a cached page from the wrong region. Your pipeline keeps running; the data is just wrong.

Verify at two levels. First, confirm your egress IP geolocates where you think, using an IP-echo endpoint fetched through the same path as your real requests. Second, and more importantly, assert on the content itself:

import re

def verify_market(page_text: str, expected_currency: str, expected_url_hint: str, final_url: str):
    checks = {
        "currency_symbol": expected_currency in page_text,
        "not_redirected_away": expected_url_hint in final_url,
        "no_region_interstitial": "choose your country" not in page_text.lower(),
    }
    failed = [k for k, ok in checks.items() if not ok]
    if failed:
        raise ValueError(f"geo verification failed: {failed}")

Wire this into the pipeline as a hard gate, the same way you'd gate on schema validation. A geo assertion failure should stop ingestion for that market, not write a row.

Two more field notes. Cache-bust while testing, because CDNs regularly serve region-mixed cached pages and you'll chase ghosts. And keep sessions market-consistent: cookies acquired through a German exit will confuse requests sent through a US one, so use one session per market. The same discipline applies if you're crawling a whole site per region: crawl each market as a fully separate pass with its own frontier, because internal links differ between regional variants too.

The Ethics and ToS Part, Briefly and Honestly

Geotargeted fetching is how legitimate businesses do price intelligence, ad verification, and localization QA. It's also capable of misuse, so a few lines I'd actually stand behind:

  • Read the site's terms. Some ToS explicitly restrict automated access or circumventing region controls. That's a business and legal decision for you, ideally with counsel if the stakes are real, not something a blog post can wave away.
  • Region-locked and licensed content is a different category. Using geo techniques to access content that is licensed per-territory (streaming media is the classic case) is a much sharper legal question than checking a public product price. I'd stay out of it.
  • Politeness doesn't change with geography. Rate limits and crawl etiquette apply from every exit node; if you're getting 429s, back off properly instead of rotating harder.
  • Public pages, honest purpose. Checking what a site publicly shows users in a market is the defensible core use case. The further you drift from that, the fewer defenses you have.

Boring advice, but the teams that follow it are the ones still running in year three.

Need to see a page the way users in another country see it? link.sc fetches any URL with geotargeting and returns clean markdown. Free tier includes 500 credits a month.