RAG vs AI Agents: Which Architecture Should Enterprises Use?
The two are usually presented as rival architectures. They are not even the same kind of thing, and treating the choice as a preference is how organisations end up with an agent where they needed a search box.
Ask which architecture an enterprise AI project should use and you will normally be offered two options: retrieval-augmented generation, or agents. Pick one.
It is a category error, and an expensive one.
RAG is how a model gets facts it was never trained on. Retrieve relevant material, put it in the context window, generate an answer grounded in it.
An agent is how a model decides what to do next. Give it tools, let it call them, feed the results back, repeat until the task is done.
These answer different questions. Most production systems worth building are an agent whose most-used tool happens to be retrieval — or a retrieval system that never needed a loop in the first place. The useful question is not which pattern, it is does this task get finished by reading, or by doing?
What each one costs you
The difference that matters operationally is bounded versus unbounded work.
A RAG query has a fixed shape: one retrieval, one generation, occasionally a rerank. Latency lands in a predictable band. Cost per query varies with context length but stays inside a range you can forecast. When it fails, it fails in one place — the wrong passages were retrieved, and the model answered confidently from them.
An agent is unbounded by construction. It runs until it decides it is finished. One request might take two model calls; another, running the same code against a harder question, might take fifteen. The cost distribution has a long tail, and teams that budget on the mean discover this at the end of the first month. When it fails, errors compound: a wrong tool call produces a wrong observation, which justifies the next wrong call.
None of this makes agents a bad idea. It makes them a thing you adopt with your eyes open, and instrument before you scale.
The decision, in three questions
- Can the task be completed from documents the person is already allowed to read? If yes, build RAG. Stop here. A large share of enterprise use cases — policy questions, procedure lookup, contract search, onboarding — are search problems with a language interface on top.
- Does completing it require changing state somewhere else? Raising a ticket, updating a record, sending something. That is a tool call, and tool calls are agent territory.
- Is the number of steps knowable in advance? If the workflow is “retrieve, then summarise,” write that as two steps. If it genuinely cannot be known until the model has seen intermediate results, you need a loop.
A blunter version that has never let us down: would a competent new joiner answer this by reading, or by doing?
The same organisation, two answers
Take two requests that arrive in the same week.
“What is our policy on contractor equipment?” — the answer exists, written down, in a document the asker is entitled to read. There is nothing to decide and nothing to change. Retrieve the passage, answer with a citation, let them open the source. Adding a reasoning loop here buys latency and cost and returns nothing.
“My laptop won’t join the VPN after the update.” — the answer does not exist anywhere yet. Getting to it means checking which build the device is on, looking at whether the certificate is current, possibly opening a ticket. The number of steps depends on what each step reveals. That is an agent, and the tools it needs are specific and few.
Same organisation, same platform, same identity model. Different architecture, because the tasks are different shapes. The mistake is not choosing wrongly between them — it is assuming the organisation has to choose once.
Where enterprises actually go wrong
Building an agent because it demos better. An agent visibly reasoning through steps is more impressive in a steering committee than a box that returns a cited paragraph. The paragraph is usually what the business asked for.
Blaming the model for a retrieval problem. When answers are poor, the instinct is to change model or rewrite the prompt. Far more often the right passage was never retrieved: chunking split it in half, the index lacks the document, or the query terms do not match the language of the source. Measure retrieval separately from generation, or you will keep tuning the wrong component.
Treating permissions as a later phase. This is the one that stops projects dead at the security review.
Having no evaluation set. Without a fixed set of questions with known-good answers, nobody can say whether last week’s change helped. Prompt changes become superstition.
The security difference nobody costs properly
Both patterns need retrieval to respect the permissions of the person asking. That means capturing access control at indexing time — the Entra ID groups permitted to see each document — and filtering on the user’s group membership at query time. An index that ignores this is a system that will cheerfully quote the redundancy plan to the person named in it.
Retrieval permissions, though, are the easy half.
Consider indirect prompt injection. A document in your index contains text that reads as an instruction rather than as content. Perhaps it arrived by email; perhaps a supplier put it in a PDF.
In a RAG system, the worst case is a bad answer. Bounded, embarrassing, recoverable.
In an agent with tools, the worst case is a bad action. The injected instruction does not need to persuade a person — it needs to persuade a model that is already permitted to call a function. If that function writes to a system, the blast radius is no longer what the model can say. It is what the tool can do.
This changes three design decisions:
- Whose identity does the tool use? Application permissions are convenient and dangerous: the agent can then do anything the application can, for anybody. Delegated permissions, acting on behalf of the signed-in user, keep the blast radius at the size of that user’s own access.
- Which tools write? Read-only tools are cheap to reason about. Every write tool deserves its own threat model, and probably its own approval gate.
- Where does a human sit? Human-in-the-loop is not a UX nicety for the first release, to be optimised away later. For irreversible actions it is the control.
The architecture most organisations should actually build
Not a choice between two things. A sequence.
Start with retrieval done properly: permission-trimmed, evaluated, with citations the user can open and check. This is unglamorous, it is most of the value, and it is the part that determines whether anyone trusts the system.
Then add tools one at a time. Each with its own permission scope, its own evaluation cases, its own decision about whether a human approves it. A “tool” here is often a single well-defined action, and a system with three good ones beats a system with twenty vague ones.
Add the loop last. Once the model can call tools reliably in a fixed sequence, letting it choose the sequence is an increment — not a rewrite. Teams that start with the loop end up debugging retrieval quality, permission design and tool reliability simultaneously, through a non-deterministic control flow, which is exactly as pleasant as it sounds.
Costing them honestly
RAG can be budgeted per query with reasonable confidence. Agents need budgeting on the tail: measure p95 model calls per task, not the average, and set a hard cap on iterations so a pathological case cannot run all afternoon.
Both benefit from the same discipline — cost telemetry attributed per use case from day one, so the question “what does this assistant cost per conversation?” has an answer that is not a shrug. A system that cannot answer it will eventually be switched off by someone in finance who could not evaluate it any other way.
What actually determines the outcome
Not the pattern.
The enterprise AI systems that survive contact with real users are the ones where somebody decided early:
- how retrieval quality is measured, separately from answer quality
- whose identity each tool call runs as
- which actions require a human, and who that human is
- what the system costs per conversation, and who owns that number
Settle those and either architecture will serve you. Leave them unsettled and the more autonomous your design, the faster it will find the gap.