Quick answer: A system prompt is the instruction you give an LLM that sets its role, rules, and behavior for an entire conversation, separate from the user's messages. The user asks questions; the system prompt defines how the assistant should answer them. It is the most powerful lever you have for shaping model behavior, and most people underuse it.
If your LLM app behaves inconsistently, ignores your formatting requirements, or wanders off task, the fix usually lives in the system prompt. Let me walk through what it is, what belongs in it, and how it works alongside tools and retrieval.
System Prompt vs User Message
Modern chat models take messages with roles. The two you care about most are system and user.
- The system prompt is standing configuration. It applies to the whole conversation and defines who the assistant is and how it should behave.
- User messages are the actual requests, one turn at a time.
The distinction matters because the model treats them differently. The system prompt carries more authority and persists across turns. A user can ask ten different questions; the system prompt shapes how all ten are answered without being repeated.
Think of it like the difference between a job description and a specific task. The job description (system prompt) says "you are a support agent who is concise, cites sources, and never guesses at policy." The task (user message) says "does my plan include priority support?" The description governs how every task is handled.
What Belongs in a System Prompt
A good system prompt covers a handful of specific things. Here is the checklist I use.
| Element | What it does | Example |
|---|---|---|
| Role | Sets the persona and domain | "You are a technical support assistant for a SaaS product." |
| Rules | Hard constraints on behavior | "Never invent pricing. If unsure, say so." |
| Format | Shape of the output | "Answer in under 100 words. Use bullet points for steps." |
| Tools | When and how to use them | "Use the search tool when the question involves current information." |
| Tone | Voice and style | "Be direct and warm. Skip filler." |
Notice what is not on the list: the user's actual question. That goes in the user message. The system prompt is about the constant rules, not the variable request.
Good vs Bad System Prompts
Here is a weak system prompt:
You are a helpful assistant.
That does almost nothing. The model falls back to generic default behavior, which means inconsistent formatting, hedging, and no domain awareness.
Here is a strong one for the same use case:
You are a support assistant for Acme, a project-management SaaS.
Rules:
- Answer only from the documentation provided in the conversation.
- If the answer is not in the provided docs, say "I don't have that in
our docs" and suggest contacting support. Never guess.
- Never state prices or plan limits unless they appear in the provided docs.
Format:
- Keep answers under 120 words.
- Use numbered steps for any procedure.
- End with a one-line "Source:" referencing the doc you used.
Tone: direct, friendly, no filler.
The difference is night and day. The second prompt constrains hallucination, sets a consistent format, and tells the model exactly when to defer. That last point (never guess, defer when unsure) is one of the highest-value instructions you can include, because it directly attacks the reasons LLMs hallucinate.
Using a System Prompt in Code
Here is a minimal example with the Anthropic SDK. The system parameter is separate from the messages array:
import anthropic
client = anthropic.Anthropic()
response = client.messages.create(
model="claude-opus-4-8",
max_tokens=1024,
system=(
"You are a support assistant for Acme, a project-management SaaS. "
"Answer only from the documentation provided in the conversation. "
"If the answer is not in the provided docs, say so and suggest "
"contacting support. Never invent prices or plan limits. "
"Keep answers under 120 words and end with a 'Source:' line."
),
messages=[
{"role": "user", "content": "How do I archive a project?"}
],
)
print(response.content[0].text)
The system string stays constant across every request in the conversation. Only the messages list changes as the user asks new things.
How the System Prompt Interacts With Retrieved Content
This is where system prompts get interesting in real applications.
When you build a retrieval-augmented app, you fetch relevant documents and put them into the conversation, usually in a user message or as tool results. The system prompt is what tells the model how to treat that content.
A retrieval system prompt should explicitly say: answer from the provided sources, do not use outside knowledge, and cite what you used. Without that instruction, the model may blend its training data with your documents and produce answers that mix fact with guesswork.
The content itself has to be clean for this to work. If you dump raw HTML into the conversation, the model wastes attention on markup and may misread the actual information. Fetching pages as clean markdown first keeps the retrieved context readable. A single call handles it:
import requests
doc = requests.post(
"https://api.link.sc/v1/fetch",
headers={"x-api-key": "lsc_your_key_here"},
json={"url": "https://acme.com/docs/archiving", "format": "markdown"},
).json()["content"]
# Then pass `doc` into the conversation, with a system prompt that says
# "answer only from the provided documentation."
Pairing a strict system prompt with clean retrieved content is the core recipe for grounding answers, which we cover in ground LLM answers in live web sources. You can see the full fetch and search API in the link.sc docs.
How the System Prompt Interacts With Tools
When your model has tools available (web search, database lookups, sending messages), the system prompt governs when it reaches for them.
Two mistakes are common. The first is being too aggressive: "ALWAYS use the search tool" makes the model call it even when it does not need to, wasting time and money. The second is being too vague, so the model never uses the tool when it should.
The fix is a clear trigger condition: "Use the search tool when the question involves recent events, current prices, or anything after your knowledge cutoff. Otherwise answer directly." That gives the model a rule it can apply consistently. For more on shaping model behavior with instructions, see what is prompt engineering.
Practical Tips
- Put constraints before formatting. Rules the model must never break should come first and be unambiguous.
- Prefer positive instructions. "Answer in under 120 words" beats "do not be verbose."
- Keep it stable. The system prompt is sent on every request, so keep it tight to save tokens, and avoid changing it mid-conversation when you can.
- Test the edges. Ask the questions your prompt is meant to handle badly (out-of-scope, missing data) and confirm the model defers instead of guessing.
The Bottom Line
A system prompt is the configuration layer of an LLM conversation. It sets the role, the hard rules, the output format, and the conditions for using tools and retrieved content. A vague one leaves the model to its defaults. A specific one gives you consistent, grounded, on-brand behavior across every turn.
If you take one thing away: the system prompt is where you tell the model to stay grounded, defer when unsure, and cite its sources. That single habit prevents most of the failures people blame on the model itself.
Feeding retrieved web content into your system prompt? link.sc returns clean markdown from any URL, so your model reads the content, not the markup. Get started free.