RAG, fine-tuning, or long context? Choose the right AI memory strategy

When builders first add AI to their product, they usually start from the same place: fire off an API call, get a response, ship it. But within a few weeks, the same question always surfaces — how do I give the AI knowledge about my product, my users, or the domain I'm building in?

Three answers show up in every discussion: RAG (retrieval-augmented generation), fine-tuning, and long context windows. Each can work. Each has failure modes. Picking the wrong one early means a painful rewrite later — picking the right one sets you up to scale without surprises.

Here's how to think through the choice.

What "memory" actually means for an LLM

An LLM has no memory between calls by default. Every request starts fresh — the only thing the model "knows" is what you put in the context window of that specific request. So when you want your AI to know something — your product documentation, a user's history, your codebase — you have to decide how to get that knowledge into the prompt.

The three strategies answer that question differently:

• RAG retrieves relevant chunks at query time and injects them into the prompt.
• Fine-tuning bakes knowledge or behavior into the model weights during a training pass.
• Long context loads a large chunk of knowledge directly into the context window with every request.

None of these is "best" in the abstract. They solve different problems.

When RAG is the right call

RAG works best when your knowledge base is large, changes frequently, or needs to be traceable. Think: product documentation, a library of past user conversations, a knowledge base your team updates every week.

The core loop is simple:

user query → embed query → retrieve top-k chunks → inject into prompt → generate response

RAG shines because the source material stays outside the model. When your docs change, you re-index them — the model stays the same. When a user asks why the AI said something, you can point to the exact chunk it retrieved.

The failure modes are retrieval quality and chunk design. If your embeddings don't surface the right passages, the model fills the gap with confident-sounding hallucinations. Garbage retrieval makes the model worse, not better.

RAG also adds latency and infrastructure: you need a vector store, an embedding pipeline, and retrieval time before you generate. For simple products, that's overhead worth auditing before you commit.

If your knowledge base changes more than once a month, or lives in documents rather than structured examples, start with RAG.

When fine-tuning is the right call

Fine-tuning teaches the model a new behavior or style, not just new facts. It's the right choice when you want the model to respond in a specific format, with a particular tone, or following domain-specific conventions that are hard to encode in a system prompt alone.

The canonical example: teaching a model to always return valid JSON in a very specific schema, using terminology particular to your domain. You can push toward this with prompting — but fine-tuning locks it in reliably at scale.

Fine-tuning is also effective when you have a large library of high-quality input-output pairs and want to distill those patterns into the base weights. Think of it like curriculum training: you're teaching the model what good looks like for your specific use case.

What fine-tuning is not: a knowledge injection tool. Fine-tuned models don't reliably retain specific facts from training data. If you fine-tune on your documentation, the model learns the shape of your responses — but it won't reliably surface individual facts you trained it on. That's RAG's job.

Fine-tuning is also the most expensive option to run correctly: you need a curated dataset, training infrastructure, evals to validate the tuned model, and a refresh process as your needs evolve. Check the Building AI Products archives for more on designing the eval layer before you invest in tuning.

If the problem is behavioral consistency — format, tone, convention — fine-tuning is worth the investment. If it's facts, use RAG.

When long context is the right call

Context windows have grown dramatically. Models now handle 200k tokens — roughly 500 pages of text — in a single call. For many products, that's enough to skip both RAG and fine-tuning entirely.

Long context is the simplest approach by far: load your knowledge directly into the prompt — a codebase, a set of documents, a full conversation history — and let the model reason over it. No retrieval pipeline, no training pass.

This works surprisingly well for:

• Code assistant tools: load the full project context into a single prompt
• Document Q&A: small document sets that fit comfortably in the window
• Conversation history: sessions short enough to keep full context

The catch is cost. Filling a 200k-token context window on every request adds up fast, even at moderate usage. Models also tend to attend better to content at the beginning and end of long contexts — so you can still get effective "retrieval failures" in the middle of a huge context.

Long context also becomes fragile at scale. When your product grows, what fit in 200k tokens won't fit in 500k, and suddenly you need RAG anyway. If you're building toward scale, plan for RAG from the start — even if you use long context as a stepping stone.

Use long context when the problem is small, stable, and cost isn't yet a constraint. Plan to graduate to RAG.

A decision framework

Signal	Likely right choice
Knowledge changes weekly or more	RAG
You need traceability ("why did it say that?")	RAG
You want consistent format / tone / style	Fine-tuning
You have 500+ quality input-output pairs	Fine-tuning
Knowledge base fits under ~100k tokens	Long context
Cost isn't a constraint yet	Long context (then RAG)
You need both facts and behavioral consistency	RAG + fine-tuning combined

What we use at Codepet

At Codepet, the AI companions need to know about a user's current project, their skill level, and the feedback history across past sessions. That's a retrieval problem — the per-user context is large, highly personalized, and updated after every session.

We use RAG to pull in the relevant user history and project context before each response. The base model handles teaching style and tone — we haven't needed fine-tuning yet because careful prompting and a detailed system prompt have been enough to establish the voice. Long context fills in where retrieval has gaps.

The pattern: RAG for facts, prompting for behavior, long context as a fallback. It's not exotic. Most production AI products converge here eventually — the question is just how many rewrites it takes to get there.

The concrete takeaway

Don't pick a memory strategy based on what sounds most sophisticated. Pick the simplest one that solves the actual problem:

• Users ask questions that need answers from your data → RAG
• You need the model to behave differently → fine-tuning
• The data fits in the context window → long context

Start simple. The right strategy usually becomes obvious when you sit with a real user and watch where the model falls down. That's also the moment when the evals you wrote earlier catch the regressions as you switch — and save you from a silent quality regression in production.