Real estate runs on data that is public, plentiful, and scattered across a dozen sites that all want you to look at it one listing at a time. Zillow, Redfin, Realtor.com, county records, rental portals, and the local MLS feed each hold a piece of the picture, and none of them hand you the whole thing in a clean file. If you are doing comps, tracking a market, or hunting for deals, the manual version of this job is hours of tab-switching that goes stale the moment you finish.
Scraping is how you collapse that into a dataset you actually own. Here is what works, where the real friction is, and the three workflows that pay off most.
What You Can Actually Pull
Almost everything an investor or analyst cares about is on a public listing page: address, list price, beds and baths, square footage, lot size, days on market, price history, and the agent or seller contact. County and city sites add the parts the portals hide: assessed value, tax history, ownership records, and permit filings. Rental portals give you asking rents by unit type, which is the other half of any cash-flow model.
The catch is not finding the data. It is that each source structures it differently, updates on its own schedule, and defends itself to a different degree. A public county assessor page is a plain HTML table you can read with anything. A major consumer portal is a heavily JavaScript-rendered app sitting behind commercial-grade bot defenses. Treating those two the same is the first mistake people make.
Why Your First Script Fails on the Big Portals
The request everyone writes first looks reasonable and returns almost nothing useful:
import requests
r = requests.get("https://www.example-portal.com/homes/123-main-st")
print(r.status_code) # 200, but the body has no listing data
You get a status that looks fine and a body that is a loading shell. The price, the price history, the beds and baths are all hydrated by JavaScript after the page loads, and a plain requests call never runs any of it. Spoofing a User-Agent does not help either, because your TLS fingerprint already announced you as a Python client. This is the layer-below-HTTP problem that trips up most scraping projects, and it is worth understanding the mechanics: see curl vs headless vs stealth browser and scrape JavaScript-rendered websites.
To read a rendered listing from these portals you need a real browser fingerprint, JavaScript execution, and often a residential egress IP so the request does not look like it came from a datacenter. Many of the biggest real estate sites also sit behind Cloudflare or similar, which adds a challenge page on top of everything else (scrape Cloudflare-protected sites covers that specific wall). County and government sites, by contrast, usually need none of this. Match the effort to the target.
Before you build anything at scale, know where the line sits. Public listing facts are one thing; republishing a portal's compiled database or ignoring its terms is another. Read is web scraping legal and keep your volume low and your purpose defensible.
A Fetch That Returns Structured Data
Rather than run and babysit a browser fleet, you can hand the URL to a fetch API that already climbs the anti-bot ladder and give it a schema describing the fields you want. link.sc renders the JavaScript, presents a real browser fingerprint, routes through residential IPs when needed, and returns exactly those fields.
import requests
schema = {
"type": "object",
"properties": {
"address": {"type": "string"},
"list_price": {"type": "number"},
"beds": {"type": "number"},
"baths": {"type": "number"},
"sqft": {"type": "number"},
"year_built": {"type": "number"},
"days_on_market": {"type": "number"},
"price_history": {"type": "array", "items": {"type": "string"}},
},
}
resp = requests.post(
"https://api.link.sc/v1/fetch",
headers={"x-api-key": "lsc_YOUR_KEY"},
json={
"url": "https://www.example-portal.com/homes/123-main-st",
"format": "json",
"schema": schema,
},
)
listing = resp.json()["data"]
print(listing["list_price"], listing["sqft"])
You get back a tidy record instead of a megabyte of app state you have to reverse-engineer. If a page has gone behind a hard block, the response tells you, rather than leaving you with a silently empty field. If you are weighing this against a licensed data feed, web scraping vs API: which to use is the right frame for that decision.
Workflow 1: Aggregating a Market Into One Dataset
The point of scraping across portals is to build the view no single site gives you. Pick a geography, collect the search-result pages for it, and fetch each listing into the schema above. Normalize everything to one shape keyed by address or parcel ID so a house that appears on three portals collapses into one record with three price sources.
Run it on a schedule and you have a living dataset: new listings, price cuts, and pending sales all show up as diffs between runs. That is the raw material for every downstream analysis, and it is yours, in a database you can query, rather than trapped in a UI.
Workflow 2: Comp Analysis That Stays Current
Comparable sales are the heart of valuation, and the manual version rots fast. With a scraped dataset, a comp query is just a filter: same submarket, similar square footage and bed count, sold in the last six months. Because you are storing price history, you can compute price per square foot, spot how far homes are selling above or below list, and track how those numbers move week to week.
Layer in county tax and assessment records and you can flag properties assessed well below recent comparable sales, which is often where the mispricing lives. The value is not any single number; it is having the whole comp set refresh automatically instead of rebuilding a spreadsheet every time a client asks.
Workflow 3: Investor Lead Generation
For investors, the dataset is a prospecting engine. The signals that predict a motivated seller are all scrapable: long days on market, repeated price reductions, expired or relisted listings, out-of-state ownership from county records, and pre-foreclosure or tax-delinquency filings where those are public. Pull the listing or owner contact where it is exposed and you have a lead with context attached.
The mechanism is a poller plus a diff, the same pattern as price monitoring. Fetch your target searches on a schedule, compare against what you saw last run, and alert on the events that matter:
import json, pathlib
seen = json.loads(pathlib.Path("seen.json").read_text() or "{}")
addr = listing["address"]
prev = seen.get(addr)
if prev is None:
notify(f"NEW: {addr} at ${listing['list_price']}")
elif listing["list_price"] < prev["list_price"]:
cut = prev["list_price"] - listing["list_price"]
notify(f"PRICE CUT: {addr} down ${cut} (DOM {listing['days_on_market']})")
seen[addr] = listing
pathlib.Path("seen.json").write_text(json.dumps(seen, indent=2))
Being first to a fresh price cut on an aging listing is the whole edge. The competitor pricing monitor post covers the value-based diffing pattern in more depth, and it maps directly onto listings. If short-term rentals are your angle, how to scrape Airbnb listings pairs naturally with this for rent estimates.
Where to Draw the Line
Keep it slow and public. County and assessor sites are open records built to be read; scrape them freely and politely. Consumer portals are stricter, so keep volume genuinely low, cache aggressively so you fetch each page once rather than hammering it, and never republish a portal's full database as if it were yours. The difference between a useful, durable pipeline and a blocked one is mostly restraint.
Real estate data is public enough that scraping it is squarely worth doing, and structured enough that once you have it in one place, market analysis and lead gen stop being a research chore and become a query you run every morning.
Point a URL at link.sc and get clean structured listing data back, with the anti-bot ladder handled for you and 500 free credits every month.