Quick answer: Normalize in a fixed order: fix encoding and whitespace first, then parse typed values (dates, prices, units) into canonical representations, then validate every record against an explicit schema with something like pydantic. Records that fail validation go to a quarantine file, not into your dataset. The schema is the contract; everything upstream of it is allowed to be messy, and nothing downstream of it is.
Scraped data is messy in ways that hand-written test data never prepares you for. Prices as "1.299,00 €", dates as "yesterday", weights as "2 lbs 3 oz", and a \xa0 lurking inside every string. I've watched pipelines silently ingest "N/A" as a product name for weeks. This post is the cleaning pipeline I actually use, step by step, with the order mattering more than people expect.
Rule Zero: Keep the Raw Data
Before any cleaning: store the raw extraction, byte for byte, separately from the normalized output. Normalization code has bugs, and requirements change. When you discover in month three that you should have kept the currency code, the raw archive means you re-run a function instead of re-scraping the web. Disk is cheap; recrawling isn't.
Everything below transforms raw into normalized. It never overwrites raw.
Step 1: Encoding and Whitespace
Boring, universal, and where every pipeline should start, because every later step assumes strings are sane.
The usual suspects: non-breaking spaces (\xa0), zero-width characters, mojibake like "’" where an apostrophe should be (a UTF-8 string decoded as Latin-1 somewhere upstream), Windows-1252 "smart quotes", and runs of whitespace where the HTML had nested tags.
import unicodedata
import re
try:
from ftfy import fix_text # pip install ftfy, worth it
except ImportError:
fix_text = lambda s: s
ZERO_WIDTH = dict.fromkeys(map(ord, ""), None)
def clean_text(s: str | None) -> str | None:
if s is None:
return None
s = fix_text(s) # repair mojibake
s = unicodedata.normalize("NFKC", s) # unify unicode forms, \xa0 -> space
s = s.translate(ZERO_WIDTH)
s = re.sub(r"\s+", " ", s).strip()
return s or None # empty string becomes None
That last line is a real decision: empty strings become None so that "missing" has exactly one representation. You do not want to check for "", "N/A", "-", and None in every downstream consumer. Map all of them to None here, once.
MISSING = {"", "n/a", "na", "none", "null", "-", "--", "tbd", "unknown"}
def null_if_missing(s: str | None) -> str | None:
if s is None or s.strip().lower() in MISSING:
return None
return s
One upstream note: a lot of whitespace and boilerplate mess never needs cleaning if the extraction is clean to begin with. Fetching pages as markdown instead of scraping raw HTML (link.sc does this) eliminates a whole class of nav-menu-in-the-product-description bugs, and asking for structured JSON directly moves even more of this work upstream.
Step 2: Dates Into ISO 8601, Always With a Sniff Test
Scraped dates arrive as "03/04/2025" (March or April, depending on the country), "2 days ago", "Yesterday", and "Mar 4th, '25". Parse them all into timezone-aware ISO 8601 and never let a locale-ambiguous string touch your database.
from datetime import datetime, timedelta, timezone
import re
from dateutil import parser as dateparser # pip install python-dateutil
def parse_date(raw: str | None, dayfirst: bool = False) -> datetime | None:
if not raw:
return None
s = raw.strip().lower()
now = datetime.now(timezone.utc)
if s in ("today",):
return now
if s in ("yesterday",):
return now - timedelta(days=1)
m = re.match(r"(\d+)\s+(minute|hour|day|week)s?\s+ago", s)
if m:
n, unit = int(m.group(1)), m.group(2)
return now - timedelta(**{unit + "s": n})
try:
dt = dateparser.parse(raw, dayfirst=dayfirst)
except (ValueError, OverflowError):
return None
if dt.tzinfo is None:
dt = dt.replace(tzinfo=timezone.utc)
return dt
The dayfirst flag is the honest part: for "03/04/2025" no parser can know the answer, only you can, based on which site the record came from. Make it explicit per-source configuration, not a global guess. And sanity-check the output: a "published date" in 1970 or 2038 is a parse failure wearing a suit.
Step 3: Prices and Units Into Numbers Plus a Code
The cardinal sin is storing "$1,299.00" as a string, or worse, stripping the symbol and storing 1299.0 with no record of the currency. A price is two fields: a decimal amount and an ISO currency code. Same logic for weights and dimensions: one number, one canonical unit.
from decimal import Decimal
import re
CURRENCY_HINTS = {"$": "USD", "€": "EUR", "£": "GBP", "¥": "JPY",
"usd": "USD", "eur": "EUR", "gbp": "GBP"}
def parse_price(raw: str | None, default_currency: str | None = None):
if not raw:
return None, None
s = raw.strip()
currency = default_currency
for hint, code in CURRENCY_HINTS.items():
if hint in s.lower():
currency = code
break
digits = re.sub(r"[^\d.,]", "", s)
# "1.299,00" -> comma is decimal; "1,299.00" -> dot is decimal
if "," in digits and "." in digits:
if digits.rfind(",") > digits.rfind("."):
digits = digits.replace(".", "").replace(",", ".")
else:
digits = digits.replace(",", "")
elif "," in digits:
digits = digits.replace(",", ".") if len(digits.split(",")[-1]) == 2 else digits.replace(",", "")
try:
return Decimal(digits), currency
except Exception:
return None, None
GRAMS = {"g": 1, "kg": 1000, "oz": 28.3495, "lb": 453.592, "lbs": 453.592}
def parse_weight_grams(raw: str | None) -> float | None:
if not raw:
return None
m = re.match(r"([\d.]+)\s*([a-z]+)", raw.strip().lower())
if not m or m.group(2) not in GRAMS:
return None
return float(m.group(1)) * GRAMS[m.group(2)]
Use Decimal for money, not float. And notice parse_price returns (None, None) on failure rather than guessing; guessing is how a "1.299" euro price becomes a $1.30 bargain in your dataset.
Step 4: Normalize Fields, Then Deduplicate on Them
Dedup belongs after normalization, because "Apple iPhone 15 Pro" and "apple iphone 15 pro" only collide once casing and whitespace are unified. Build a canonical key from the fields that define identity and keep the most complete record per key. (Cross-page dedup during the crawl itself is its own topic; this is about field-level dedup within your extracted records.)
def dedup_key(record: dict) -> str:
name = (record.get("name") or "").lower()
name = re.sub(r"[^a-z0-9]+", " ", name).strip()
return f"{name}|{record.get('sku') or ''}"
Step 5: Validate With a Schema, Quarantine the Rest
This is the step that turns "a folder of JSON" into a dataset. Define the shape you promised your consumers, in code, with pydantic, and refuse to emit anything that doesn't conform.
from datetime import datetime
from decimal import Decimal
from pydantic import BaseModel, HttpUrl, field_validator
class Product(BaseModel):
name: str
url: HttpUrl
price: Decimal | None = None
currency: str | None = None
weight_g: float | None = None
published: datetime | None = None
in_stock: bool | None = None # tri-state: True / False / unknown
@field_validator("name")
@classmethod
def name_sane(cls, v: str) -> str:
if not (2 <= len(v) <= 300):
raise ValueError("name length out of range")
return v
@field_validator("price")
@classmethod
def price_sane(cls, v):
if v is not None and not (0 < v < 1_000_000):
raise ValueError("price out of plausible range")
return v
def validate_batch(records: list[dict]):
good, quarantined = [], []
for r in records:
try:
good.append(Product(**r))
except Exception as e:
quarantined.append({"record": r, "error": str(e)})
return good, quarantined
Two opinions embedded in that model. First, optional fields are explicitly | None, which forces every downstream consumer to acknowledge that the value might be missing; that honesty beats backfilling fake defaults like price=0. Second, validators check plausibility, not just type: a price of 950000 parses fine as a Decimal and is almost certainly a cents-vs-dollars bug.
Quarantined records go to a JSONL file you actually review. In my experience the quarantine rate is the best single health metric a scraping pipeline has: a jump from 1% to 20% overnight means the site changed its layout, and you found out from your own pipeline instead of an angry user.
The Order Is the Pipeline
Recap, because the sequencing is the actual lesson: raw archive, then text cleaning, then typed parsing (dates, money, units), then dedup, then schema validation with quarantine. Each stage assumes the previous one ran, which is exactly why fixing encoding after parsing prices, or deduping before normalizing case, produces the flaky pipelines everyone complains about.
Messy input is a given. A messy dataset is a choice.
Skip the messiest step: link.sc extracts clean markdown or structured JSON from any URL, so your pipeline starts from sane input. 500 free credits a month.