← All posts

How to Find the Hidden API Endpoints Behind a Web Page

GUIDES · JUL 18, 2026 · 6 MIN READ

Quick answer: Open your browser DevTools, go to the Network tab, filter to XHR/Fetch, and reload the page. Most modern sites load their data from a JSON API in the background, and that request shows up in the list. Click it, read the response, copy the URL and headers, and you can often pull the same structured data directly instead of parsing HTML. This is cleaner than scraping the rendered page, but only do it for public data and within the site's terms of service.

Almost every site you would want to scrape today is really two things: a shell of HTML and a pile of JavaScript that fetches the actual data from an API. When you scrape the HTML, you are reading the output of that process. When you find the API, you read the input. The input is smaller, structured, and far more stable.

Here is how to find it and how to use it responsibly.

Why the API is usually cleaner than the HTML

HTML is a presentation format. Class names change when a designer reworks the layout, content gets wrapped in extra divs, and the same data appears in three places for responsive breakpoints. A scraper built on CSS selectors breaks the week the marketing team ships a redesign.

The underlying API returns JSON that maps directly to the site's data model. Field names like price, sku, and availability change far less often than the markup around them, because backend contracts are expensive to break.

HTML scraping JSON API
Format Markup you must parse Structured data
Stability Breaks on redesigns Breaks only on API changes
Payload size Full page Just the data
Pagination Guess from links Explicit params
Rendering needed Often (for SPAs) No

If the site is a single-page app and you have been fighting to get the rendered HTML after JavaScript runs, finding the API is often the escape hatch. You skip the browser entirely.

Step 1: Open the Network tab and reload

In Chrome or Firefox, open DevTools (F12 or right-click and choose Inspect), then click the Network tab. Reload the page with the Network tab open so it captures every request from the start.

You will see dozens of entries: the HTML document, CSS, fonts, images, analytics beacons, and somewhere in there, the data calls.

Step 2: Filter to XHR and Fetch

Click the Fetch/XHR filter at the top of the Network panel. This hides static assets and leaves the requests JavaScript made for data. That is where the API calls live.

Now interact with the page the way a user would. Scroll to trigger infinite loading, click into the next page of results, expand a section. Each action that pulls new content fires a request you can watch appear in real time. That is how you map which endpoint serves which piece of data.

Step 3: Read the request and response

Click a promising entry. The panel splits into tabs:

  • Headers shows the full request URL, method, query string, and every header the browser sent.
  • Payload (or Request) shows the body for POST requests.
  • Preview and Response show what came back, usually JSON you can expand and browse.

Look at the Preview tab first. If you see the data you wanted as a clean tree of fields, you found the right call. Note the URL, the method, and any query parameters that look like they control pagination or filtering (page, offset, limit, cursor, sort).

Step 4: Replicate the request

The fastest way to reproduce a request is to let the browser write it for you. Right-click the entry and choose Copy as cURL. Paste that into your terminal and you have an exact, working replica, including every header.

curl 'https://example.com/api/products?category=shoes&page=2' \
  -H 'accept: application/json' \
  -H 'user-agent: Mozilla/5.0 ...' \
  -H 'x-requested-with: XMLHttpRequest'

Then trim it. Remove headers one at a time and re-run until it stops working. Most requests need only a few: accept, sometimes user-agent, and occasionally a specific header the API checks. Once you know the minimum set, translate it into your language of choice.

import requests

params = {"category": "shoes", "page": 2}
headers = {"accept": "application/json"}

resp = requests.get(
    "https://example.com/api/products",
    params=params,
    headers=headers,
    timeout=20,
)
data = resp.json()

for item in data["products"]:
    print(item["name"], item["price"])

Pagination becomes a loop over the page or cursor param instead of a fragile walk through "next" links in the HTML.

When the request needs more than headers

Some endpoints sign their requests with a token minted by the page's JavaScript, or set a cookie the API validates. If a request only works with a short-lived token or an authenticated session, that is the site signaling the data is not meant for anonymous automated access. Treat that as a boundary, not a puzzle to solve.

If you have a legitimate, authorized reason to access data behind a session (your own account, or explicit permission), the mechanics of session handling are covered separately in our guide to scraping data behind a login. Do not use tokens or sessions to reach data a normal anonymous visitor cannot see.

When this is appropriate, and when it is not

Finding an internal API is a technique, and like any technique it is neutral. The ethics live in how you use it.

It is generally fine when:

  • The data is public, visible to any visitor without logging in.
  • You respect the site's terms of service and robots.txt.
  • You keep request rates polite, roughly matching human browsing, and cache aggressively so you do not hammer the endpoint.

It is not appropriate when:

  • The endpoint requires authentication and you are reaching data you are not entitled to.
  • The terms of service prohibit automated access and you are bound by them.
  • You would be scraping personal data in a way that violates privacy law.

A hidden API is often more efficient for the server than repeated HTML scraping, because you request less and parse nothing on their end. But "more efficient" is not "unlimited." Rate-limit yourself, identify your bot honestly where a policy asks you to, and back off on errors.

A one-call alternative

Sometimes you do not want to babysit headers, tokens, and rotating layouts across a hundred sites. That is the case link.sc is built for: point link.sc at a URL and it returns clean, structured content, handling rendering, proxies, and parsing for you.

curl https://link.sc/v1/fetch \
  -H "Authorization: Bearer lsc_..." \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com/products", "format": "markdown"}'

For sites where you have found a clean public JSON API, calling it directly is the lightest option. For everything else, or when you want one consistent interface across many sources, an API like link.sc saves the maintenance. Use whichever keeps your pipeline honest and stable.

The Network tab is one of the most underused tools in scraping. Spend five minutes there before you write a single selector, and you will often find you did not need selectors at all.

Want structured web data without reverse-engineering a new API for every site? Try link.sc free.