Quick answer: The compliant way to get Twitter (now X) data is the official X API, not an HTML scraper. It returns posts, users, and engagement metrics as structured JSON across paid access tiers. Scraping X's pages directly violates its Terms of Service, and X has actively cracked down on automated access. The platform has changed a lot since 2023, so verify current tiers and limits before you build. Reach for the API first.
Twitter, rebranded as X, is a magnet for scraping interest because the public conversation is valuable for research, sentiment analysis, and trend tracking. It is also one of the most volatile targets in this space: access rules, API tiers, and enforcement have all shifted dramatically in the last couple of years. So the honest answer starts with a caveat: check the current documentation, because anything specific I state could be outdated by the time you read it.
Start with the X API
X runs an official API that exposes the data you would otherwise be tempted to scrape. It is the sanctioned path, which makes it both the compliant option and the only one X clearly permits.
What it broadly covers:
- Posts (tweets): text, author, timestamps, and engagement metrics like likes, reposts, and replies.
- Users: profile fields and public account metrics.
- Search and timelines: recent and, on higher tiers, fuller historical access.
- Streaming (filtered): real-time delivery of posts matching your rules, on tiers that include it.
Responses are structured JSON, so you skip HTML parsing entirely.
The API tiers, qualitatively
X restructured its API into paid access tiers. Rather than quote prices or caps that change (and I will not invent numbers), here is the shape you should expect and then confirm in the current docs:
- A free or low-cost entry tier with tight limits, oriented toward testing and small write-focused use.
- Mid tiers aimed at startups and developers, with larger monthly post-retrieval caps and more endpoints.
- Enterprise access for high-volume, historical, and firehose-style needs.
The important part: pricing and limits have moved more than once, so treat any figure you read (here or elsewhere) as needing verification against the official pricing page before you commit.
API vs scraping the HTML
| X API | Scraping HTML | |
|---|---|---|
| Terms of service | Sanctioned path | Prohibited |
| Data format | Structured JSON | Markup and embedded JSON to parse |
| Stability | Documented, versioned | Breaks constantly, heavy blocking |
| Rate limits | Defined per tier | Undocumented, aggressive enforcement |
| Historical data | Available on higher tiers | Unreliable |
| Risk | Low, within terms | ToS breach, account and IP bans |
What is and is not public
X has increasingly gated content that used to be openly viewable. The line matters:
- Public: posts and profiles that render to a logged-out visitor, where X still allows that.
- Not public: anything requiring a login to view, and anything behind rate-limited or gated views.
If your access depends on logging in, accepting the terms, or getting past a limit or block, you have left "public data" and should stop.
Compliant approaches
- Use the X API tier that matches your volume, and design queries to fit its limits.
- Cache and deduplicate so you are not re-pulling the same posts and burning your cap.
- Consider licensed data providers with their own agreements if you need scale beyond what your tier allows (evaluate their compliance posture).
- Narrow your scope: most projects need a specific set of accounts or a keyword stream, not the whole platform.
A code sketch
A minimal recent-search call against the X API v2 (bearer token from your developer account):
import requests
BEARER = "YOUR_X_API_BEARER_TOKEN"
resp = requests.get(
"https://api.twitter.com/2/tweets/search/recent",
headers={"Authorization": f"Bearer {BEARER}"},
params={
"query": "from:example -is:retweet",
"max_results": 25,
"tweet.fields": "created_at,public_metrics",
},
timeout=30,
)
resp.raise_for_status()
for post in resp.json().get("data", []):
print(post["created_at"], post["public_metrics"]["like_count"], post["text"][:80])
No headless browser, no proxy rotation, no fragile selectors. You request fields and get typed JSON back, within your tier's limits.
Fetching public pages you are entitled to read
For the narrow, legitimate case of reading a single public page you can access without logging in (for example, a public post embedded or linked elsewhere, or your own site's content), link.sc returns clean markdown or JSON in one call:
curl https://link.sc/v1/fetch \
-H "Authorization: Bearer lsc_..." \
-H "Content-Type: application/json" \
-d '{
"url": "https://www.example.com/public-post",
"format": "markdown"
}'
Use link.sc for the legitimate slice: public pages you can access and official API responses. It is not a tool for bypassing X's login or anti-bot defenses, and you should not use any tool that way.
The legal and ethics note
X's Terms of Service prohibit scraping and automated access outside the API, and X has pursued action against scrapers and bulk crawlers. Circumventing login or technical access controls can raise CFAA exposure in the US. Posts and profiles contain personal data, so the GDPR and similar laws govern how you collect, store, and use them, and "it was public" is not by itself a lawful basis under the GDPR.
Guardrails:
- Use the X API at the tier that fits, and verify current limits and pricing yourself.
- Never bypass login, rate limits, or anti-bot systems.
- Handle personal data lawfully, with a basis for storage and use.
- Talk to counsel for anything at scale. None of this is legal advice.
For the broader framing, read is web scraping legal and ethical web scraping and compliance best practices.
The bottom line
X is the volatile case: the rules and tiers have changed repeatedly, and scraping the site is both prohibited and heavily enforced. The compliant path is the official API at a tier matched to your volume, careful query design, and licensed providers if you need more scale. Verify the current tiers and limits before you build, because whatever you remember from a year ago is probably out of date.
Need clean content from public pages and official API responses? link.sc fetches any public URL to markdown or JSON in one call. Try it free.