Quick answer: curl is the fastest way to fetch a page or hit an API from the command line. curl https://example.com grabs the HTML, -L follows redirects, -H sets headers, -o saves output, and -d sends a POST body. It is perfect for static pages and JSON APIs. It cannot run JavaScript or defeat anti-bot systems, so for rendered pages or blocked sites you point curl at a fetch API instead, which does the rendering and proxying and returns clean content.
curl is on nearly every machine, it speaks HTTP fluently, and it is the quickest way to see what a server actually returns before you write a line of scraper code. I reach for it constantly to debug, to prototype, and to script small jobs. Let me cover the flags that matter, then be honest about the ceiling.
The basics: GET a page
The simplest possible fetch prints the response body to your terminal:
curl https://example.com
That is the raw HTML the server sent. To make it useful you usually add a few flags.
# -s silences the progress meter, -L follows redirects.
curl -sL https://example.com
Without -L, curl stops at the first 301 or 302 and prints nothing useful. Most real sites redirect (http to https, apex to www), so -L is almost always what you want when scraping.
Setting headers
Many servers filter the default curl User-Agent, so set a realistic one. Use -H for any header, and repeat it for several.
curl -sL \
-H "User-Agent: Mozilla/5.0 (compatible; MyScraper/1.0)" \
-H "Accept-Language: en-US,en;q=0.9" \
https://example.com
To send cookies or an auth token, headers are again the tool:
curl -sL \
-H "Authorization: Bearer some-token" \
-H "Cookie: session=abc123" \
https://example.com/dashboard
Saving output
Piping HTML to your screen gets old. Save it to a file with -o (you choose the name) or -O (uses the remote filename).
# Save to a chosen filename.
curl -sL https://example.com -o page.html
# Save using the server's filename.
curl -sLO https://example.com/report.pdf
To inspect only the status and headers without the body, use -I for a HEAD request, or -i to include headers alongside the body:
curl -sIL https://example.com # headers only
curl -sw "%{http_code}\n" -o /dev/null https://example.com # just the status code
That last one is handy in scripts: it discards the body and prints the numeric status, so you can branch on whether a page is reachable.
POSTing to an API
Scraping is not only HTML. A lot of data lives behind JSON endpoints, and curl posts to them cleanly with -d (or --data). Set the content type so the server parses your body.
curl -sL -X POST https://api.example.com/search \
-H "Content-Type: application/json" \
-d '{"query":"web scraping","limit":10}'
If a page loads its data from a background JSON call (open your browser's network tab to find it), hitting that endpoint directly with curl is often cleaner and more stable than parsing rendered HTML.
A quick reference
| Flag | What it does |
|---|---|
-s |
Silent: hide the progress meter |
-L |
Follow redirects |
-H |
Set a request header (repeatable) |
-o / -O |
Save output to a file |
-I |
HEAD request, headers only |
-i |
Include response headers with the body |
-X |
Set the HTTP method (GET, POST, ...) |
-d |
Send a request body |
-w |
Write out a custom format, like %{http_code} |
When curl is enough, and when it is not
curl is the right tool when the page returns real HTML in the initial response, or when the data lives behind a JSON API you can call directly. It is fast, scriptable, and has no dependencies.
curl is not the right tool in two situations. First, JavaScript rendering: curl fetches the HTML the server sends and does not execute scripts, so a single-page app hands you an empty shell where the content should be. Second, anti-bot systems: curl requests from a datacenter IP get flagged, and you land on CAPTCHAs or 403s that no header tweak resolves. For the full breakdown of these tradeoffs, see curl vs headless vs stealth browser.
A note on etiquette while we are here: scrape only public data, honor robots.txt, keep your request rate reasonable, and do not use headers to impersonate a logged-in user you are not. Legitimate access to public information is the goal.
Pointing curl at a fetch API
When you hit the rendering or blocking wall, you do not have to abandon curl. You point it at a fetch API that does the rendering and proxying server-side and returns clean content. Here is a call to the linksc fetch endpoint. The auth header is x-api-key, never Authorization: Bearer:
curl -sL -X POST https://api.link.sc/v1/fetch \
-H "x-api-key: lsc_your_api_key" \
-H "Content-Type: application/json" \
-d '{"url":"https://example.com","format":"markdown"}'
The response is JSON shaped like {"content":"..."}, where content is the page as clean markdown, with the JavaScript already rendered and proxies already handled. Pipe it through jq to pull the content field out:
curl -sL -X POST https://api.link.sc/v1/fetch \
-H "x-api-key: lsc_your_api_key" \
-H "Content-Type: application/json" \
-d '{"url":"https://example.com","format":"markdown"}' \
| jq -r '.content'
Search is the same pattern, and the search field is q:
curl -sL -X POST https://api.link.sc/v1/search \
-H "x-api-key: lsc_your_api_key" \
-H "Content-Type: application/json" \
-d '{"q":"curl web scraping","engine":"google"}'
That returns {"serpData":{...}} with full page content, not just the ten blue links. If you want to understand what a fetch API is doing behind that one curl call, what is a web scraping API explains it.
The bottom line
Keep curl in your toolkit for exactly what it is good at: quick fetches, header debugging, and calling JSON APIs. When you hit JavaScript rendering or blocking, the smallest change is to keep using curl but point it at an API that handles those problems, so your shell scripts and cron jobs stay simple.
You can start on the free tier (500 credits per month) and check the details at link.sc/pricing.
Want to fetch any URL as clean markdown with a single curl call? Get a free link.sc API key and try it in your terminal.