Quick answer: Fine tuning is the process of taking a pretrained language model and training it further on your own examples so it adapts its behavior, style, or format to a specific task. It changes the model's weights, unlike prompting or retrieval, which leave the model untouched. Fine tuning is powerful for shaping how a model responds, but it is a poor fit for teaching new facts, and for most teams retrieval is the cheaper and faster first move.
What Fine Tuning Actually Is
A base model like GPT or Claude has already seen a huge slice of the public internet during pretraining. Fine tuning takes that generalist and trains it on a smaller, curated dataset of your own input-output pairs. The model updates its internal weights to match the patterns in your examples.
Think of it as on-the-job training for a model that already speaks the language. You are not teaching it English from scratch. You are teaching it to answer the way your support team answers, to output JSON in your exact schema, or to write in your brand voice consistently.
The key distinction: fine tuning changes the model. Prompting and retrieval do not. When you write a clever prompt or paste in some context, the underlying weights are identical before and after. When you fine tune, you produce a new set of weights that behaves differently by default.
How Fine Tuning Works: Full vs LoRA
There are two broad approaches, and the difference matters a lot for cost.
Full fine tuning updates every weight in the model. You run your dataset through the model, compute how wrong its outputs are, and adjust all of the parameters through backpropagation. This is the most thorough method and also the most expensive. For a large model you need serious GPU memory, because you have to hold the model, the gradients, and the optimizer state all at once.
Parameter-efficient fine tuning (PEFT), of which LoRA is the most common form, freezes the original weights and trains a small set of new parameters instead. LoRA (Low-Rank Adaptation) inserts tiny trainable matrices into the model and only updates those. The frozen base stays intact, and the adapter you train is often a few megabytes rather than gigabytes.
Here is the practical comparison:
| Aspect | Full fine tuning | LoRA / PEFT |
|---|---|---|
| Weights updated | All of them | A small added set |
| GPU memory needed | High | Much lower |
| Output size | A full model copy | A small adapter file |
| Swap between tasks | Load a whole model | Swap the adapter |
| Risk of forgetting | Higher | Lower |
| Typical use | Deep behavior change | Most task adaptation |
For the vast majority of teams, LoRA or a similar PEFT method is the right choice. It captures most of the benefit at a fraction of the memory and storage cost, and you can keep several task-specific adapters around and swap them without shipping multiple full models.
What Fine Tuning Is Good At
Fine tuning shines when you want to change how a model behaves rather than what it knows.
- Consistent format. If you need reliable JSON, a specific XML shape, or a fixed structure every time, fine tuning on examples teaches the model that shape far more reliably than prompt instructions alone.
- Tone and style. Brand voice, a legal register, a terse support style. These are patterns the model can absorb from a few hundred good examples.
- Narrow, repetitive tasks. Classification, routing, extraction, and rewriting into a house style all fine tune well because the task is stable and the outputs are predictable.
- Latency and cost at scale. A fine tuned smaller model can sometimes match a larger prompted model on a narrow task, which lowers per-call cost once you are running high volume.
What Fine Tuning Is Bad At
This is where people burn time and money, so it is worth being blunt.
Fine tuning is a weak way to teach a model facts. If your goal is for the model to know your latest product docs, your pricing, or events from last week, fine tuning is the wrong tool. Facts baked into weights are hard to update, easy to get wrong, and prone to being remembered fuzzily. The model may confidently blend your data into a plausible but incorrect answer, which is exactly the hallucination problem you were trying to avoid.
Fine tuning also does not handle freshness. The moment your data changes, your fine tuned model is stale, and retraining is not free. Anything that updates daily or weekly is a bad candidate.
And it needs data and iteration. You need a clean dataset of examples, a way to evaluate the result, and usually several rounds before it is good. If you do not have a few hundred high-quality examples, you do not have enough to fine tune well.
Why RAG Is Usually the Better First Move
When people say "we need to fine tune the model on our data," they almost always mean "we need the model to answer questions using our data." Those are different problems, and the second one is a retrieval problem.
Retrieval-augmented generation (RAG) keeps the model as-is and instead fetches the relevant documents at query time and puts them in the prompt. The model reads your data fresh on every request. Update a document and the next answer reflects it immediately, no retraining required. You also get citations, because you know which source you handed the model.
For a deeper side-by-side on when each approach wins, see our RAG vs fine tuning breakdown. The short version:
| Need | Reach for |
|---|---|
| Model should know your changing facts | RAG |
| Answers must cite sources | RAG |
| Data updates often | RAG |
| Fixed output format every time | Fine tuning |
| Specific tone or style | Fine tuning |
| Narrow high-volume task, lower cost | Fine tuning |
| Both behavior and knowledge | RAG first, then fine tune if needed |
My rule of thumb: start with a good prompt, add RAG when the model needs your knowledge, and only fine tune once you have a stable task and evidence that prompting plus retrieval has hit a ceiling. Fine tuning is a scalpel, not a starting point.
Where the Data Comes From
Both fine tuning and RAG live or die on data quality, and a lot of that data lives on the open web: docs, help centers, product pages, and reference material. Getting that content into clean, model-ready text is its own chore, because most pages are heavy HTML wrapped in navigation and scripts.
That is the gap link.sc fills. You point it at a URL and get back clean markdown ready to chunk for a retrieval index or to shape into fine tuning examples:
curl https://api.link.sc/v1/fetch \
-H "x-api-key: lsc_your_key" \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com/docs/pricing", "format": "markdown"}'
The response comes back as { "content": "..." } with the readable text extracted and the page furniture stripped out. Whether you are building a retrieval corpus or curating fine tuning pairs, starting from clean text saves you a preprocessing headache. You can read more in the link.sc docs.
The Bottom Line
Fine tuning changes the model to shape how it behaves. It is excellent for format, tone, and narrow repetitive tasks, and it is genuinely bad at teaching facts or keeping up with fresh data. For most real-world "use our data" projects, retrieval gets you there faster, cheaper, and with citations. Reach for fine tuning when you have a stable task, real examples, and evidence that prompting and RAG have run out of room.
Building a data pipeline for RAG or fine tuning? Get clean, model-ready web content from any URL with link.sc.