Quick answer: Build on official data first. Most real estate portals prohibit scraping in their terms, and the cleaner path is a licensed feed: RESO Web API from an MLS, a partner or affiliate feed, or a public open-data source. Where you do pull from public pages, extract a normalized set of listing fields, dedupe the same property across portals with a fingerprint, geocode addresses, and refresh on a schedule so stale and sold listings drop out.
An aggregator lives or dies on two things: whether the data is allowed to be there, and whether the same house shows up three times because three portals list it. Get those right and the rest is plumbing.
Get your sources right before you write any code
This is the part people skip, and it is the part that sinks projects. Real estate data is heavily governed, and the good sources are not the ones that scrape easiest.
| Source | Access | Data quality | Terms |
|---|---|---|---|
| MLS via RESO Web API | Licensed, needs broker sponsorship | Highest, authoritative | Governed by MLS rules |
| Partner / affiliate feed | Approved program | High | Per-agreement |
| Public open data (county, gov) | Open | Varies | Usually permissive |
| Consumer portals | Public pages | Good, but restricted | Most prohibit scraping |
Prefer the top of that table. The RESO Web API is a standardized way MLSs expose listings, and if you have a broker relationship it is the authoritative feed everyone else is a copy of. Many large portals also run partner programs that hand you a sanctioned feed instead of making you scrape.
Where you genuinely need public-page data, read the terms of service and robots.txt for each site, pull only public listing pages, keep your rate polite, and never touch anything behind a login. If a site says no in its terms, believe it and find another source. Ethical web scraping compliance best practices goes deeper on drawing that line.
The listing fields worth normalizing
Every portal names things differently. Decide on your canonical schema early so ingestion from any source maps into the same shape.
{
"source": "county-open-data",
"source_id": "R-2291845",
"url": "https://...",
"address": "123 Maple St, Austin, TX 78704",
"lat": 30.245,
"lng": -97.765,
"price_usd": 549000,
"beds": 3,
"baths": 2,
"sqft": 1680,
"lot_sqft": 6100,
"property_type": "single_family",
"status": "active",
"listed_date": "2026-07-02",
"photos": ["https://..."],
"fingerprint": "…",
"last_seen": "2026-07-18T00:00:00Z"
}
The two fields that do the heavy lifting later are fingerprint (for dedupe) and last_seen (for freshness). Everything else is display data.
Extracting fields from public pages
When a source is a public HTML page rather than a feed, fetch it to clean text and extract against your schema. Raw HTML is full of galleries, maps, and mortgage-calculator widgets that waste tokens and confuse a parser, so render to markdown first.
curl https://link.sc/v1/fetch \
-H "Authorization: Bearer lsc_..." \
-H "Content-Type: application/json" \
-d '{
"url": "https://open.county.gov/listings/R-2291845",
"format": "markdown"
}'
From clean markdown, a single schema-bound extraction pass fills the canonical object. The mechanics of that (fixed fields, null contract, JSON out) are the same as in extracting structured JSON from any webpage, so I will not repeat them here.
Dedupe across portals
This is the hard problem. The same property appears under different IDs, formatted addresses, and even slightly different square footage. You need a fingerprint that is stable across those variations.
import hashlib, re
def normalize_addr(addr: str) -> str:
addr = addr.lower()
addr = re.sub(r"\b(street|st|avenue|ave|road|rd|drive|dr)\b", "", addr)
addr = re.sub(r"[^a-z0-9]", "", addr)
return addr
def fingerprint(listing: dict) -> str:
# Round coordinates so tiny geocoding differences still collide.
key = "|".join([
normalize_addr(listing["address"]),
f"{round(listing['lat'], 4)}",
f"{round(listing['lng'], 4)}",
str(listing.get("beds", "")),
])
return hashlib.sha256(key.encode()).hexdigest()[:16]
Fingerprint on normalized address plus rounded coordinates plus bed count. When two listings collide, merge them: keep the richest field set, list all source URLs, and prefer the most authoritative source for price and status. Coordinate rounding matters because two geocoders will place the same house a few meters apart.
Geocoding and mapping
Addresses come in as strings; a map needs coordinates. Run every incoming address through a geocoder once and cache the result keyed by the raw address string, because geocoding is the slowest and often the metered step in the pipeline. Cache aggressively and you geocode each unique address exactly once.
Store the coordinates on the listing and you can serve map bounds queries directly from your database instead of geocoding on the fly.
Keeping it fresh
A listings aggregator that shows sold houses is worse than no aggregator. Freshness is a feature.
- Re-crawl on a schedule sized to each source. Active markets churn daily; open-data dumps update weekly.
- Track
last_seen. Every crawl updates it. If a listing has not been seen in N crawls, mark itstale, thenremoved. - Watch status transitions. Active to pending to sold is the signal your users care about most.
Detecting when a listing changes (price drop, status flip) is a change-monitoring problem. The pattern of fetch, diff, alert is covered in monitor web page changes and get alerts, and it maps cleanly onto re-crawls: diff the new fetch against the stored listing and emit an event on any meaningful field change.
Storage
A relational store fits this well because the queries are relational: filter by price and beds within a map bounding box, ordered by listed date. Put an index on (lat, lng) or use a spatial extension (PostGIS) for real geo queries, a unique constraint on fingerprint so dedupe is enforced at write time, and an upsert on ingest so re-crawls update in place rather than piling up duplicates.
INSERT INTO listings (fingerprint, source, price_usd, status, last_seen)
VALUES (%(fingerprint)s, %(source)s, %(price)s, %(status)s, now())
ON CONFLICT (fingerprint)
DO UPDATE SET price_usd = EXCLUDED.price_usd,
status = EXCLUDED.status,
last_seen = now();
An honest caution
The technical parts here are straightforward. The part that will actually determine whether your project survives is the sourcing. Consumer real estate portals invest heavily in their listing data and most explicitly forbid redistribution, so scraping them into a competing aggregator is both a terms violation and a legal risk. That is not a scare tactic; it is the reason licensed feeds exist. Build on a feed you are allowed to use, treat public open data as public, and keep humans in the loop on anything ambiguous.
Do it that way and you get an aggregator that does not evaporate the first time a portal's legal team sends a letter.
Aggregating listings from public sources? link.sc turns any listing page into clean markdown or schema-bound JSON, so you spend your time on dedupe and freshness, not parsing. Start free.