Quick answer: Start with structured sources before you scrape anything. For U.S. public companies, SEC EDGAR exposes filings as machine-readable data (XBRL financial facts and a JSON API), so you rarely need to parse HTML by hand. For everything else (investor-relations pages, PDF earnings decks, press releases), fetch the page to clean text, then run one LLM extraction pass to pull the metrics into a fixed JSON schema you control.
Financial data is messy in a specific way: the numbers are precise, but where they live is not. Revenue might sit in an XBRL tag, a PDF table, a press-release bullet, or a slide. This post walks the order I actually use, cheapest and most reliable first.
Where financial data lives, and how hard each source is
Not all sources are equal. Reach for the structured ones before you touch a scraper.
| Source | Format | Structured? | Effort |
|---|---|---|---|
| SEC EDGAR company facts API | JSON (XBRL) | Yes | Low |
| SEC EDGAR full-text filings | HTML / iXBRL | Partly | Medium |
| Investor-relations pages | HTML | No | Medium |
| Earnings press releases | HTML | No | Medium |
| Earnings decks / 10-K PDFs | No | High |
The rule of thumb: if the company files with the SEC, EDGAR already did most of the work. If you are pulling from a marketing IR page or a PDF deck, you are extracting from prose, and that is where an LLM pass earns its keep.
Use EDGAR's structured data first
The SEC publishes filing data as JSON. The company facts endpoint returns every XBRL fact a company has ever reported, keyed by the standard US-GAAP tags. No scraping, no parsing, and it is free.
You need the company's CIK (a zero-padded 10-digit identifier). The SEC keeps a lookup at https://www.sec.gov/files/company_tickers.json. Once you have it, one request gets you the facts.
import requests
# SEC asks that you send a descriptive User-Agent with contact info.
headers = {"User-Agent": "MyCompany Research [email protected]"}
cik = "0000320193" # Apple Inc., zero-padded to 10 digits
url = f"https://data.sec.gov/api/xbrl/companyconcept/CIK{cik}/us-gaap/Revenues.json"
data = requests.get(url, headers=headers).json()
# Each unit entry is a reported value with its period and source filing.
for row in data["units"]["USD"]:
if row.get("form") == "10-K":
print(row["end"], row["val"], row["accn"])
That gives you audited numbers straight from the filing, with the period and the accession number so you can trace every value back to its source. When a structured tag exists, this beats any HTML parse for accuracy.
When you have to parse the page
Structured data covers the big, tagged line items. It does not cover an investor-relations landing page, a segment breakdown buried in a press release, or a guidance number a CFO put in a slide. For those, fetch the page as clean text first.
Feeding raw HTML to an extraction model wastes tokens and confuses it with nav bars and cookie banners. link.sc's fetch endpoint renders the page (including JavaScript-heavy IR portals) and returns clean markdown, so the model sees the numbers, not the chrome.
curl https://link.sc/v1/fetch \
-H "Authorization: Bearer lsc_..." \
-H "Content-Type: application/json" \
-d '{
"url": "https://investor.example.com/news/q2-2026-results",
"format": "markdown"
}'
The same endpoint handles PDF filings. Point it at the URL of a 10-K or an earnings deck and it returns the text content, tables included, without you standing up a PDF pipeline.
Structure it with an LLM extraction pass
Once you have clean text, the extraction step is a single model call with a fixed schema. Do not ask for "the important numbers." Ask for named fields with types and a null contract, so the output is predictable enough to store.
import anthropic, json
client = anthropic.Anthropic()
schema = {
"period": "string, e.g. Q2 2026",
"revenue_usd": "number or null",
"net_income_usd": "number or null",
"eps_diluted": "number or null",
"gross_margin_pct": "number or null",
"guidance_next_quarter_revenue_usd": "number or null",
}
prompt = f"""Extract these fields from the earnings text below.
Return ONLY valid JSON matching this schema. Use null for anything
not explicitly stated. Do not calculate or infer values.
Schema: {json.dumps(schema)}
Text:
{clean_markdown}
"""
resp = client.messages.create(
model="claude-sonnet-4-6",
max_tokens=1024,
messages=[{"role": "user", "content": prompt}],
)
result = json.loads(resp.content[0].text)
Two rules keep this honest. First, tell the model to use null rather than guess, so a missing number stays missing instead of becoming a hallucinated one. Second, tell it not to calculate: if you want a margin the release did not print, compute it yourself in code where you can audit the arithmetic. LLMs are good at finding a stated number in prose and bad at being a spreadsheet.
If you want the whole thing in one hop, link.sc can return JSON directly against a schema you pass in, so the fetch and the extraction collapse into a single request. That pattern is covered in extracting structured JSON from any webpage.
Validate before you trust
Financial data has cheap sanity checks. Use them.
- Cross-check against EDGAR when a filing exists. If your parsed revenue disagrees with the XBRL fact, trust XBRL and flag the parse.
- Range-check every value. Negative revenue, an EPS of 4,000, or a margin above 100% is a parse error, not news.
- Keep the source. Store the filing URL and accession number alongside every number so any figure is one click from its origin.
- Watch units. "Millions" in a table header is the most common source of a value that is off by a factor of a million. Confirm the multiplier per source.
A compliance note
Extracting financial data comes with real obligations, and they are worth stating plainly.
SEC EDGAR is public and built to be consumed by machines. The SEC does ask that automated access stays under about 10 requests per second and sends a descriptive User-Agent with contact info. Respect both. For company IR pages, check robots.txt and terms of use, pull only public pages, and keep your request rate polite.
Then there is what you do with the numbers. Do not trade or advise on data before it is public, do not restate figures as fact without tracing them to a filing, and remember that an extraction pipeline is a research aid, not a substitute for reading the actual 10-K. If you build tooling that others rely on for decisions, an LLM parse should surface the source, not hide it. For a broader treatment of the legal side, see is web scraping legal.
Putting it together
The pipeline that holds up over time is boring on purpose: EDGAR JSON for anything tagged, fetch-to-markdown for pages and PDFs, one schema-bound LLM pass for the prose, and validation against the structured source. Cheapest and most reliable steps first, the fragile HTML parse last and always cross-checked.
Get that order right and you spend your time on analysis instead of babysitting a scraper that breaks every time an IR team redesigns their site.
Building a financial-data pipeline? link.sc turns any filing, IR page, or PDF into clean markdown or schema-bound JSON in one call. Start free.