Quick answer: For Ruby web scraping the classic stack is Net::HTTP from the standard library to fetch a page and Nokogiri to parse it with CSS selectors or XPath. That handles static HTML cleanly. When the page renders content with JavaScript or blocks your IP, install the linksc gem (gem "linksc") and call a fetch API that does the rendering, proxying, and markdown conversion for you. Both approaches are shown below with runnable code.
Ruby has a lovely story for HTML parsing. Nokogiri is fast, its selector API reads naturally, and it has been the default for over a decade. If your targets are server-rendered pages, you can build a clean scraper in a dozen lines. The friction, as always, shows up when the page needs a browser to render or when the site actively blocks bots.
Here is the from-scratch path first, then the gem, with honest guidance on when to reach for each.
The from-scratch approach: Net::HTTP + Nokogiri
Nokogiri is the only dependency you need for parsing. Net::HTTP ships with Ruby.
gem install nokogiri
A minimal scraper that fetches a page and extracts links:
require "net/http"
require "uri"
require "nokogiri"
uri = URI("https://example.com")
# Net::HTTP with a sane timeout and a real User-Agent.
http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = uri.scheme == "https"
http.open_timeout = 10
http.read_timeout = 15
request = Net::HTTP::Get.new(uri)
request["User-Agent"] = "Mozilla/5.0 (compatible; MyScraper/1.0)"
response = http.request(request)
unless response.is_a?(Net::HTTPSuccess)
abort "Unexpected status: #{response.code}"
end
doc = Nokogiri::HTML(response.body)
doc.css("a").each do |link|
href = link["href"]
next unless href
puts "#{link.text.strip} -> #{href}"
end
The pieces that matter: set both open_timeout and read_timeout so a slow host cannot hang your process, set a User-Agent because the default one gets filtered, and check for Net::HTTPSuccess before you parse. Trying to run selectors over a 403 error page wastes time and produces confusing results.
Nokogiri's selector API is where the joy is. Both CSS and XPath work:
# CSS selectors
titles = doc.css("h2.title").map(&:text)
# XPath when you need structure CSS cannot express
prices = doc.xpath('//span[@class="price"]/text()').map(&:to_s)
# Attributes and nested lookups
doc.css("article").each do |article|
heading = article.at_css("h3")&.text
link = article.at_css("a")&.[]("href")
puts "#{heading}: #{link}"
end
at_css and at_xpath return the first match (or nil), which pairs well with Ruby's safe navigation operator &. so a missing node does not raise.
Being a good citizen
A short note before scaling up. Only scrape public data, read the site's robots.txt, and keep your request rate reasonable. If you are scraping many pages, add a small sleep between requests and back off when you see 429 responses. The goal is legitimate access to public information, not evading authentication or overwhelming a server. That distinction keeps you on the right side of both etiquette and terms of service.
Where the from-scratch approach breaks
Two problems tend to end the hand-rolled approach. The first is JavaScript. Net::HTTP returns the raw HTML the server sent, and a single-page app assembles its content in the browser after load. You get an empty container where the data should be. Ruby's answer is usually a headless browser through ferrum or selenium-webdriver, which means running and managing Chrome.
The second is blocking. Requests from cloud datacenter IPs get flagged, and you hit CAPTCHAs or 403s that a header change will not solve. The fix is rotating residential proxies, which is another moving part to run. We cover this tradeoff in depth in curl vs headless vs stealth browser.
Once you are maintaining a browser pool and proxy rotation, you are running scraping infrastructure. That is when an API starts to look cheap.
The low-effort path: the linksc gem
When you would rather not run Chrome and proxies, a fetch API handles rendering, proxying, and HTML-to-markdown conversion on the server and returns clean content. The linksc gem gives you a synchronous client that fits ordinary Ruby scripts and Rails jobs.
# Gemfile
gem "linksc"
bundle install
require "linksc"
# Falls back to ENV["LINKSC_API_KEY"] if you construct without an argument.
client = Linksc::Client.new(api_key: "lsc_your_api_key")
# Fetch any URL as clean markdown. JavaScript is rendered server-side.
content = client.fetch("https://example.com", format: "markdown")
puts content
Search returns full page content rather than a list of snippets, which is what you want when you are feeding an LLM:
serp = client.search("ruby web scraping", engine: "google")
puts serp
Prefer no gem at all? The API is plain HTTP, so Net::HTTP calls it directly. The auth header is x-api-key, never Authorization: Bearer:
require "net/http"
require "json"
require "uri"
uri = URI("https://api.link.sc/v1/fetch")
request = Net::HTTP::Post.new(uri)
request["x-api-key"] = "lsc_your_api_key"
request["Content-Type"] = "application/json"
request.body = JSON.dump(url: "https://example.com", format: "markdown")
response = Net::HTTP.start(uri.host, uri.port, use_ssl: true) do |http|
http.request(request)
end
data = JSON.parse(response.body)
puts data["content"]
If you want to understand what a fetch API actually does under the hood, what is a web scraping API breaks it down.
Which should you use?
| Factor | Net::HTTP + Nokogiri | linksc gem |
|---|---|---|
| Static HTML pages | Ideal, zero API cost | Works, mild overkill |
| JavaScript-rendered pages | Needs Ferrum or Selenium | Handled server-side |
| Blocked / anti-bot sites | You run proxies | Handled for you |
| Output format | Raw HTML, you parse | Clean markdown or JSON |
| Ongoing maintenance | All yours | Mostly the API's problem |
| Cost | Compute only | Credit-based, free tier to start |
My honest recommendation: for a few static, well-behaved sites, Nokogiri is the right tool and adding an external dependency would be overengineering. Use it. When you are chasing JavaScript rendering or fighting blocks across many sources, the maintenance burden outweighs the API credits, and the gem lets you delete a lot of fragile code.
You can start free with 500 credits per month and check the details at link.sc/pricing.
Building a Ruby app or Rails job that needs clean web data? Grab a free link.sc API key and call fetch or search with the gem.