[ BLOG ]// NOTES FROM THE CRAWL
Engineering, product, and the open web
What we are building, what broke, and what we learned indexing the web for agents.
->How to Scrape a Website That Requires JavaScript to RenderWhy plain HTTP returns an empty shell on SPAs, how headless browsers fix it, and how link.sc renders JavaScript automatically with smart waits.::MCP Explained, and How to Give Claude and Cursor Web ToolsWhat the Model Context Protocol is, why it matters for web access, and how to add link.sc's fetch and search tools to Claude, Cursor, and Windsurf.</>The Best Way to Convert HTML to Markdown for LLMsWhy raw HTML wastes tokens and hurts retrieval, what to strip vs keep, the limits of Readability and Turndown, and how to get clean LLM-ready markdown.[ ]How to Scrape a Site Protected by CloudflareHow Cloudflare bot detection works, why naive requests get a 403, the escalation ladder for reliable access, and how to stay polite and ethical.##curl_cffi vs. Headless Browser vs. Stealth Browser: Which Should You Use?A practical guide to the three tiers of web fetching — plain HTTP, headless browsers, and stealth browsers — and how to pick the right one per site.$_How to Build a RAG Pipeline That Uses Live Web DataA practical walkthrough of the fetch, clean, chunk, embed, retrieve, generate loop, why clean markdown matters for retrieval, and how to keep the index fresh./*How to Do Real-Time Web Search for Your LLM AppA practical guide to giving LLMs live web access — why snippet-only search APIs fall short, and how to build a search-read-answer flow with citations.{ }How to Monitor a Competitor's Pricing Page for ChangesA practical guide to reliably tracking competitor pricing changes without drowning in false positives from JS rendering, ads, and layout noise.!=Is Web Scraping Legal, and Do I Have to Obey robots.txt?A balanced, plain-English look at when web scraping is legal, what robots.txt actually means, and the ethical practices that keep you out of trouble.@_What Really Happens When ChatGPT Searches the WebChatGPT doesn't search for what you ask. It decomposes your question into multiple sub-queries. Here's exactly how the fan-out pipeline works.>>How to See What ChatGPT, Perplexity, and Google AI Actually Search ForA practical guide to monitoring and understanding AI search behavior — from Chrome extensions to API-based approaches and building your own monitoring.&&AI Search Is a Black Box (And That Should Bother You)AI search tools decide what information billions see, but none of them show their work. The transparency gap is a problem for developers, publishers, and users alike.->AI Search Engines Are Confidently Wrong 60% of the TimeNew research reveals alarming citation accuracy failures across ChatGPT, Perplexity, and other AI search tools. Here's what developers need to know.::Want AI to Cite Your Content? You Need to Understand Fan-Out QueriesAI search tools don't search for what users ask. They decompose questions into sub-queries. Here's how to optimize your content for this new reality.</>What is /llms.txt and Why Should You Care?A plain-English breakdown of the llms.txt file — what it does, where it came from, and whether it actually matters for your website in 2026.[ ]llms.txt vs robots.txt: What's Actually DifferentThey both live in your root directory and talk to bots. But llms.txt and robots.txt solve completely different problems. Here's a clear comparison.##How to Create Your First llms.txt File (Step by Step)A practical, no-fluff walkthrough for creating an llms.txt file for your website. Covers the spec, common mistakes, and real examples you can copy.$_Does llms.txt Actually Work? What the Data SaysAn honest look at the real-world evidence for llms.txt effectiveness. We dug into the studies, the log data, and the expert opinions so you don't have to./*What is Model Context Protocol (MCP) and Why It Matters for AIAn introduction to the Model Context Protocol (MCP), how it works, and why it's becoming essential for connecting AI models to real-world data.{ }llms.txt Best Practices: Lessons from Real ImplementationsWhat Cloudflare, Stripe, and Anthropic get right (and wrong) with their llms.txt files. Practical best practices from analyzing real-world implementations.!=llms.txt and Generative Engine Optimization: The New SEO FrontierHow llms.txt fits into the bigger picture of Generative Engine Optimization (GEO) — the emerging practice of optimizing content for AI-powered search engines.@_llms.txt for API Documentation: Helping AI Understand Your ProductAPI docs are the perfect use case for llms.txt. Here's how to structure your file so AI assistants can actually help developers use your product.>>How AI Crawlers Read Your Website (And How llms.txt Fits In)A look under the hood at how GPTBot, ClaudeBot, and other AI crawlers actually process your website — and where llms.txt enters the picture.&&Why Your llms.txt File Shouldn't Be a SitemapThe most common llms.txt mistake is treating it like a sitemap. Here's why that defeats the purpose and what to do instead.->The Future of llms.txt: Where This Standard Is HeadingPredictions for where the llms.txt specification goes from here — including the scenarios where it thrives, the ones where it fades, and what replaces it if it does.::10 Common Questions About llms.txt, Answered HonestlyEvery question you actually have about llms.txt — from "do I really need this?" to "does Google care?" — answered without the marketing fluff.</>Ethical Web Scraping: Compliance and Best Practices in 2026A comprehensive guide to ethical web scraping practices, legal compliance, and responsible data collection for AI applications.[ ]Building RAG Pipelines with Real-Time Web DataLearn how to build production-grade RAG pipelines that ground LLM responses with fresh web data using link.sc.##The Ultimate Guide to SERP Scraping in 2026Everything you need to know about scraping search engine results pages in 2026, from techniques to tools to legal considerations.$_How AI Web Scraping is Transforming Data Collection in 2026Explore how AI-powered web scraping is revolutionizing data collection, from intelligent content extraction to automated pipeline optimization./*Token Optimization: How to Feed Web Data to LLMs EfficientlyStrategies for minimizing token usage when feeding web data to LLMs, from content extraction to intelligent chunking.{ }Welcome to the link.sc BlogInsights on AI web scraping, LLM data pipelines, and the future of web data.!=Web Scraping for SEO: Extract People Also Ask Questions at ScaleHow to extract and analyze Google's People Also Ask data to discover content opportunities and improve your SEO strategy.@_Getting Started with the link.sc API: A Developer's GuideA step-by-step guide to getting started with the link.sc web data API, from your first API call to production deployment.