Quick answer: A market research tool works in four moves: define the questions you want answered, gather sources from the live web (search plus fetch), extract structured signals from those sources (pricing, features, sentiment, trends), and synthesize the findings into a report with an LLM. Keep it fresh by rerunning on a schedule. The architecture is simple. The value comes from asking sharp questions and pulling from current sources instead of stale ones.
The failure mode of most "AI research" tools is that they ask a model what it already knows. That gives you a confident summary of the training data, which for anything market-related is months out of date. A real tool retrieves current pages first, then reasons over them.
Architecture
Questions -> Gather (search + fetch) -> Extract signals -> Synthesize report
^ |
|____________________ rerun on a schedule ___________________________|
Four stages, one loop. Let me walk each one.
Stage 1: Define the Questions
A research tool is only as good as its questions. Vague inputs ("tell me about the market") produce vague reports. Concrete questions produce structured answers.
QUESTIONS = [
"What do the top competitors in project management software charge per seat?",
"Which features are they highlighting on their pricing pages?",
"What are recent reviews saying about ease of use?",
"What pricing or feature changes shipped in the last quarter?",
]
Notice these map to signal types: pricing, features, sentiment, and trends. That mapping drives the extraction stage.
Stage 2: Gather Sources
For each question, search the web and fetch the promising results as clean markdown. The contract: search takes q, fetch takes url and format, both go to api.link.sc/v1 with an x-api-key header, fetch returns {"content": "..."}.
import os
import requests
LINKSC = "https://api.link.sc/v1"
HEADERS = {
"x-api-key": os.environ["LINKSC_API_KEY"],
"content-type": "application/json",
}
def search(query: str) -> list[dict]:
r = requests.post(
f"{LINKSC}/search",
headers=HEADERS,
json={"q": query, "engine": "google"},
)
r.raise_for_status()
return r.json()["serpData"].get("organic", [])
def fetch(url: str) -> str:
r = requests.post(
f"{LINKSC}/fetch",
headers=HEADERS,
json={"url": url, "format": "markdown"},
)
r.raise_for_status()
return r.json()["content"]
def gather(question: str, limit: int = 4) -> list[dict]:
sources = []
for item in search(question)[:limit]:
try:
sources.append({
"url": item["link"],
"title": item.get("title", ""),
"content": fetch(item["link"]),
})
except requests.HTTPError:
continue
return sources
Using link.sc for this step means you skip the parts that usually eat a research project: rendering JavaScript pages, rotating proxies, and parsing messy HTML. You get readable text and can spend your time on the analysis.
A quick note on scope: gather public information only, respect rate limits and robots.txt, and do not evade logins or paywalls. Public pricing pages, docs, and reviews are the raw material of legitimate market research.
Stage 3: Extract Structured Signals
Raw page text is not a signal. You want typed data you can compare across competitors. Ask the model to pull specific fields as JSON.
import json
from anthropic import Anthropic
claude = Anthropic() # reads ANTHROPIC_API_KEY
def extract_signals(source: dict) -> dict:
resp = claude.messages.create(
model="claude-opus-4-8",
max_tokens=1500,
thinking={"type": "adaptive"},
system=(
"Extract market signals from the page as JSON with keys: "
"vendor, pricing (list of {plan, price, unit}), "
"key_features (list), sentiment (positive|mixed|negative|unknown), "
"notable_changes (list). Use null or empty lists when a signal "
"is not present. Do not invent data."
),
messages=[{
"role": "user",
"content": f"URL: {source['url']}\n\n{source['content'][:8000]}",
}],
)
body = next(b.text for b in resp.content if b.type == "text")
data = json.loads(body)
data["source_url"] = source["url"]
return data
"Do not invent data" and explicit "use null when not present" are load-bearing. Without them, a model will happily fill a pricing field with a plausible-looking number it made up. You want gaps reported as gaps.
Stage 4: Synthesize the Report
Now feed all the extracted signals back to the model and ask for a report. Because it is working from structured data you retrieved, it stays grounded.
def synthesize(topic: str, signals: list[dict]) -> str:
resp = claude.messages.create(
model="claude-opus-4-8",
max_tokens=4000,
thinking={"type": "adaptive"},
system=(
"You are a market analyst. Write a concise report from the "
"structured signals provided. Compare vendors on price and "
"features in a table. Cite the source_url for every claim. "
"Flag anything the data does not cover rather than guessing."
),
messages=[{
"role": "user",
"content": f"Topic: {topic}\n\nSignals:\n{json.dumps(signals, indent=2)}",
}],
)
return next(b.text for b in resp.content if b.type == "text")
The full run:
def research(topic: str, questions: list[str]) -> str:
signals = []
for q in questions:
for source in gather(q):
signals.append(extract_signals(source))
return synthesize(topic, signals)
Stage 5: Keep It Fresh
Markets move. A report from three months ago is a historical document, not intelligence. The cheapest way to stay current is to rerun the whole pipeline on a schedule (a nightly or weekly cron job) and diff the new signals against the last run. Store each run's extracted signals with a timestamp, and you get change tracking for free: a competitor's price drop or a new feature shows up as a delta.
from datetime import date
def snapshot(topic, signals):
return {"date": str(date.today()), "topic": topic, "signals": signals}
Compare today's snapshot to last week's and surface only what changed. That turns a one-off report into a living monitor.
Where the Value Actually Is
The code above is a weekend build. What separates a useful market research tool from a toy is three things:
- Question quality. Sharp, specific questions get specific answers. Invest here.
- Source freshness. Live retrieval beats model memory every time for anything market-related. This is the whole reason the tool exists.
- Honest extraction. A tool that says "no pricing found on this page" is more valuable than one that invents a number. Reward the gaps.
Everything else is plumbing, and the plumbing is largely handled for you when search and fetch return clean, structured content.
This is one application of a broader pattern. For the competitor-tracking angle specifically, see competitive intelligence with web scraping. If you want to push the synthesis further into an autonomous loop, build a deep research agent covers that. The link.sc docs have the full API reference.
Want to build a market research tool on live web data? Start with link.sc for free.