Quick answer: Write a plain Python function that POSTs a URL to a fetch API and returns the page as markdown, then register it with @tool (LangChain) or FunctionTool.from_defaults (LlamaIndex). The agent will call it whenever it decides it needs a page. The two things that actually determine whether this works well are the tool's docstring and how aggressively you truncate the output.
I have watched a lot of people wire up their first web tool, and the wiring is never the hard part. The hard part is that a naive fetch tool returns 80,000 tokens of raw page and your agent falls over. So this tutorial covers the wiring and the parts that go wrong.
The fetch function
One plain function, shared by both frameworks. It calls link.sc's fetch endpoint, which handles JavaScript rendering, proxies, and anti-bot escalation server-side and returns clean markdown, so the function stays small:
import os
import requests
LINKSC_KEY = os.environ["LINKSC_API_KEY"] # lsc_...
MAX_CHARS = 16000 # ~4k tokens; tune per model
def linksc_fetch(url: str) -> str:
"""Fetch a web page and return its main content as clean markdown."""
r = requests.post(
"https://link.sc/v1/fetch",
headers={
"Authorization": f"Bearer {LINKSC_KEY}",
"Content-Type": "application/json",
},
json={"url": url, "format": "markdown"},
timeout=60,
)
r.raise_for_status()
content = r.json()["content"]
if len(content) > MAX_CHARS:
content = content[:MAX_CHARS] + "\n\n[truncated]"
return content
Two deliberate choices here. Markdown output, because it is a fraction of the tokens of raw HTML and models read it better (more on that in HTML to markdown for LLMs). And a hard character cap, because the single most common failure mode is an unbounded tool response blowing the context window. We will come back to that.
You could point this at raw requests.get instead of an API and it will work right up until the first JavaScript-rendered page or Cloudflare wall, which in practice is about the third URL your agent tries.
Registering it with LangChain
LangChain turns any typed, docstringed function into a tool with the @tool decorator. The docstring is not decoration: it is the text the model reads when deciding whether to call your tool, so write it like an instruction.
from langchain_core.tools import tool
from langchain_core.prompts import ChatPromptTemplate
from langchain.agents import create_tool_calling_agent, AgentExecutor
from langchain_openai import ChatOpenAI
@tool
def fetch_page(url: str) -> str:
"""Fetch a specific web page and return its content as markdown.
Use this when the user provides a URL, or when you need the actual
current content of a known page. Do not use it to search: it takes
exact URLs only.
"""
return linksc_fetch(url)
llm = ChatOpenAI(model="gpt-4o-mini")
prompt = ChatPromptTemplate.from_messages([
("system",
"You are a research assistant. Fetch pages only when you need "
"facts you don't have. After fetching, answer from the page and "
"cite the URL."),
("human", "{input}"),
("placeholder", "{agent_scratchpad}"),
])
agent = create_tool_calling_agent(llm, [fetch_page], prompt)
executor = AgentExecutor(agent=agent, tools=[fetch_page], verbose=True)
result = executor.invoke({
"input": "Read https://link.sc/pricing and summarize the plans."
})
print(result["output"])
Run it with verbose=True the first few times. Watching the scratchpad teaches you more about your agent's fetch behavior than any amount of docs.
The same tool in LlamaIndex
LlamaIndex wraps plain functions with FunctionTool. Same function, same principles, different wrapper:
from llama_index.core.tools import FunctionTool
from llama_index.core.agent import ReActAgent
from llama_index.llms.openai import OpenAI
fetch_tool = FunctionTool.from_defaults(
fn=linksc_fetch,
name="fetch_page",
description=(
"Fetch a specific web page by exact URL and return its content "
"as markdown. Use when the user gives a URL or you need the "
"current content of a known page. Not a search tool."
),
)
agent = ReActAgent.from_tools(
[fetch_tool],
llm=OpenAI(model="gpt-4o-mini"),
verbose=True,
)
print(agent.chat("What does https://link.sc/docs/quickstart say about auth?"))
ReAct agents narrate their reasoning ("I need to fetch this page..."), which makes them pleasant to debug. If you later move to LlamaIndex's workflow-based agents, the tool definition carries over unchanged; only the orchestration around it differs.
Teaching the agent when to fetch
An agent with a fetch tool has two new ways to be wrong: fetching when it should not, and not fetching when it should. You steer both with the docstring and the system prompt.
Say what the tool is not. "Not a search tool: takes exact URLs only" prevents the classic mistake where the model passes "best python http libraries" as a URL. If you want the agent to discover pages too, give it a second tool backed by the search endpoint rather than overloading fetch (the same pattern I describe in giving your AI agent internet access).
Anchor fetching to need, not curiosity. Prompts like "fetch only when you need facts you don't have" reduce reflexive fetching. Without that nudge, some models fetch a URL merely because one appears in the conversation.
Tell it what to do after fetching. "Answer from the page and cite the URL" closes the loop. Otherwise you sometimes get an agent that fetches a page and then answers from its training data anyway, which is maddening to spot.
Cap the loop. Set max_iterations (LangChain) or the equivalent so a confused agent cannot fetch fifteen pages chasing its own tail. Three to five tool calls is plenty for most tasks.
Failure modes you will hit
Token blowups. The big one. A single unbounded page can be bigger than your entire context budget, and the cost hits you twice: once in this turn, and again every turn after, because tool outputs live in the agent's scratchpad history. The MAX_CHARS cap is the blunt fix. Smarter versions: summarize long pages with a cheap model before returning them, or return the first N chars plus a table of section headings so the agent can ask for a specific section.
Fetching too much. Give an agent a tool and it will use it. If you see five fetches where one would do, tighten the system prompt and lower the iteration cap. For research-shaped tasks, consider a search tool that returns full page content in one call, so the agent does not need a fetch-per-result loop at all.
Silent tool errors. If the fetch raises and your framework swallows the exception into a string, the model may treat the error text as page content. Return errors in an unmistakable shape, like "ERROR: fetch failed for {url}: {reason}", and tell the model in the prompt what to do when it sees one (usually: tell the user, do not retry more than once).
Stale assumptions about structure. Agents sometimes fetch a page expecting a pricing table and get a marketing page. Markdown output helps here because headings survive conversion, and the model can navigate by them instead of guessing.
The part that actually matters
The framework glue is twenty lines either way, and you will not think about it again. What you will keep tuning is the trio of docstring, truncation, and system prompt, because together they decide whether the tool makes your agent smarter or just more expensive.
Start with the cap low (4k tokens of page content answers most questions), watch the verbose traces, and raise it only when you see real answers getting truncated. The free tier of 500 credits a month is enough to run these examples and a lot of experimentation besides.
Give your agent a fetch tool that survives real-world websites: get a free link.sc API key and paste it into the code above.