← All posts

Sentiment Analysis of Web Data: A Practical Guide

GUIDES · JUL 18, 2026 · 5 MIN READ

Quick answer: Sentiment analysis of web data means collecting public text (reviews, news articles, forum posts), cleaning it into plain sentences, scoring each piece as positive, negative, or neutral, and aggregating the scores into a trend you can watch over time. You have two scoring engines to choose from: a fast lexicon that counts sentiment words, or an LLM that reads context. The hard part is rarely the scoring. It is collecting clean text and knowing what the numbers actually mean.

Most tutorials hand you a movie-review dataset and a classifier and call it done. Real sentiment work starts one step earlier: the text lives on the open web, in HTML, behind pagination, mixed with navigation and ads. Getting it into a clean, analyzable form is where the project lives or dies.

The Pipeline

Stage Goal Tool
Collect Pull public text from review sites, news, forums Search and fetch
Clean Strip HTML, boilerplate, and noise; split into units Markdown output plus light parsing
Score Assign sentiment to each unit Lexicon or LLM
Aggregate Roll scores up by source, topic, or day Simple stats
Track Store results and compare over time A dated table

Collect: Get Clean Text From the Web

Start with search to find relevant pages, then fetch each one as markdown. Because fetch returns clean content instead of raw HTML, most of the cleaning is already done for you. Note the contract: search uses q, fetch uses url and format, both hit api.link.sc/v1 with an x-api-key header.

import os
import requests

LINKSC = "https://api.link.sc/v1"
HEADERS = {
    "x-api-key": os.environ["LINKSC_API_KEY"],
    "content-type": "application/json",
}


def search(query: str) -> list[dict]:
    r = requests.post(
        f"{LINKSC}/search",
        headers=HEADERS,
        json={"q": query, "engine": "google"},
    )
    r.raise_for_status()
    return r.json()["serpData"].get("organic", [])


def fetch(url: str) -> str:
    r = requests.post(
        f"{LINKSC}/fetch",
        headers=HEADERS,
        json={"url": url, "format": "markdown"},
    )
    r.raise_for_status()
    return r.json()["content"]


def collect(topic: str, limit: int = 10) -> list[dict]:
    docs = []
    for item in search(f"{topic} reviews")[:limit]:
        try:
            docs.append({"url": item["link"], "text": fetch(item["link"])})
        except requests.HTTPError:
            continue
    return docs

A compliance note before you scale this up: collect only public data, respect robots.txt and rate limits, and do not scrape behind logins or evade authentication. Public reviews and news are fair game for analysis; a private social feed is not. When in doubt, use official APIs for platforms that offer them, and keep your request rate polite.

Clean: Split Into Scorable Units

Markdown is already free of tags, but you still want to break it into sentences or short paragraphs. Scoring a whole 2,000-word article as one number hides everything interesting inside it.

import re


def to_units(text: str) -> list[str]:
    # drop markdown headings, links, and list markers
    text = re.sub(r"[#>*_`\[\]()]", " ", text)
    # split on sentence boundaries
    parts = re.split(r"(?<=[.!?])\s+", text)
    return [p.strip() for p in parts if len(p.strip()) > 20]

Score: Lexicon vs LLM

Here is the real decision. The two approaches trade speed for understanding.

Lexicon (e.g. VADER) LLM (Claude)
Speed Very fast, local Slower, one API call per batch
Cost Free Per-token
Context Word counting only Reads meaning
Sarcasm Misses it Often catches it
Domain terms Needs a custom lexicon Adapts from the prompt
Best for High volume, rough signal Nuance, mixed sentiment, short runs

A lexicon scores in microseconds and is perfectly fine for a first pass over thousands of units. Reach for an LLM when the text is subtle, sarcastic, or full of domain language a generic word list does not know.

The lexicon path:

from nltk.sentiment import SentimentIntensityAnalyzer

sia = SentimentIntensityAnalyzer()


def score_lexicon(unit: str) -> float:
    return sia.polarity_scores(unit)["compound"]  # -1.0 to 1.0

The LLM path, batched so you do not pay for one call per sentence:

import json
from anthropic import Anthropic

claude = Anthropic()


def score_llm(units: list[str]) -> list[dict]:
    numbered = "\n".join(f"{i}: {u}" for i, u in enumerate(units))
    resp = claude.messages.create(
        model="claude-opus-4-8",
        max_tokens=2048,
        thinking={"type": "adaptive"},
        system=(
            "Score each numbered line for sentiment. Return a JSON array of "
            "objects with keys: index, label (positive|negative|neutral), "
            "score (-1.0 to 1.0). Account for sarcasm and context."
        ),
        messages=[{"role": "user", "content": numbered}],
    )
    body = next(b.text for b in resp.content if b.type == "text")
    return json.loads(body)

Aggregate and Track

Individual scores are noise. The signal is in the rollup: average sentiment per source, per topic, and per day.

from datetime import date
from statistics import mean


def aggregate(scored: list[float]) -> dict:
    if not scored:
        return {"date": str(date.today()), "n": 0, "avg": None}
    return {
        "date": str(date.today()),
        "n": len(scored),
        "avg": round(mean(scored), 3),
        "pct_positive": round(sum(s > 0.05 for s in scored) / len(scored), 3),
    }

Store each day's aggregate in a table with the date. Over weeks you get a trend line: is sentiment about your product climbing after a release, or sliding after an outage? That comparison, not any single day's number, is the point. Run the collection on a schedule so the trend keeps building on its own.

Pitfalls to Respect

Sentiment analysis fails in predictable ways. Name them up front so you do not over-trust the output.

  • Sarcasm. "Great, another update that breaks everything" is negative, but a lexicon reads "great" and scores it positive. LLMs handle this better, though not perfectly.
  • Context and negation. "Not bad at all" is positive. "I wanted to love it" leans negative. Word-counting misses both.
  • Mixed sentiment. One review can praise the battery and pan the screen. Forcing a single label throws away half the meaning; scoring at the sentence level preserves it.
  • Domain language. In finance, "aggressive" can be neutral. In gaming, "sick" is praise. Generic lexicons get these backward.
  • Sampling bias. People with strong feelings write reviews. The loudest text is not the average opinion, so treat volume and sentiment as separate signals.

None of these are reasons to skip sentiment analysis. They are reasons to report it with error bars instead of false precision. Show the trend, show the sample size, and let a human read the outliers.

If you want to turn this into an always-on monitor, the same collect-clean-score loop underpins monitoring brand mentions online. And if you plan to search the web continuously rather than in batches, real-time web search for LLMs covers the freshness side. The link.sc docs have the full search and fetch reference.

Want clean, structured text from any page for your sentiment pipeline? Start with link.sc for free.