Most teams add a vector database before they even check if they need one.
I've seen this happen a dozen times. Your AI feature needs to "know stuff." Someone says RAG. Someone else drops Pinecone or pgvector in the chat. Next thing you know you're chunking docs, generating embeddings, tuning a reranker, and babysitting an index that goes stale every other week. It ships. It works. But now you've got a second system to keep alive, and nobody asked if the simple version would've been fine.
This isn't a "RAG is dead" post. Vector DBs are the right call for a real set of problems. The issue is they've become the default answer for every AI feature. And defaults are expensive.
So here's the stuff nobody tells you before you commit.
Read article on "Rust Ownership Explained Like You're 5"
1. A Vector DB Solves One Problem. Just One.
RAG exists to solve a narrow thing: getting a model to answer using info it wasn't trained on and can't hold in context while being able to point to where the answer came from.
That's it. That's the whole job.
It's not a "make my agent smarter" button. If your agent gives bad answers because it reasons poorly over info it already has, retrieval won't save you. That's a prompting and evals problem. Bolting a vector store on top of a reasoning bug just gives you a faster way to fetch the wrong context.
So before you buy infra, name the failure. Is it "the model doesn't know this fact"? Retrieval helps. Is it "the model knows the fact but uses it badly"? Retrieval does nothing.
Read article on "Deno Deploy to Cloudflare Workers"
2. Your SaaS Data Probably Doesn't Need Similarity Search
Here's the test that kills most vector DB projects: is your data already in a database?
Users, accounts, orders, tickets, audit logs that's structured data. You don't need fuzzy semantic search to find a user. You need a tool call that runs SELECT * FROM users WHERE id = ?.
Tool calls beat vector retrieval here on three things:
Precision. The model handles a clean record way better than a chunk of prose.
Freshness. Data is correct the second it's written. No embedding pipeline to re-run.
Security. You inherit your existing access controls, transactions, and audit trail for free.
A parallel vector store sitting next to your DB has none of that by default. It's got its own staleness problem, its own permissions, its own weird failure modes.
Jason Liu said it well: "Good search is the ceiling on your RAG quality. If recall is poor, no prompt engineering or model upgrade will save you."
That cuts both ways. If SQL already solves your search, RAG adds nothing.
Read article on "OpenCode vs Claude Code"
3. Context Windows Changed Everything
RAG was born when models had 8K to 32K context windows. You had to retrieve. There was no room.
That's not true anymore. Frontier models ship 200K to 1M token windows now, and prompt caching made long contexts way cheaper than they used to be.
For static stuff product docs, a legal corpus, a codebase it's often simpler and more reliable to just dump it all in the prompt. No chunker. No embeddings. No index to keep fresh.
Iddo Gino from Datawizz said it straight: "RAG is probably no longer necessary for 95% of AI apps. The simplest approach is often the best for most apps, you can get away with sticking all the relevant info in the context window."
Anthropic's own guidance says the same thing. From Building Effective Agents: "Start with simple prompts, optimize them with comprehensive evaluation, and add multi-step agentic systems only when simpler solutions fall short."
But here's the honest part. Big contexts aren't magic either. Chroma ran a study in July 2025 across 18 frontier models and found performance drops as input gets longer way before you hit the max. A 200K model can start degrading around 50K tokens. People call this context rot.
So "dump everything in" isn't a universal fix either. The real point: you have two good options now. You used to have one.
Read article on "What No One Tells You About Real ROI"
4. File-Based Memory Covers Most Agent State
There's a middle path between "vector DB" and "dump it all in the prompt." Anthropic ships it officially.
Their Memory tool writes to a filesystem a /memories folder not a vector store. Their docs call it "the key primitive for just-in-time context retrieval." The agent saves what it learns to files and reads them back when needed.
Claude Code uses this in production. It keeps a MEMORY.md index file (roughly 200 lines or 25KB max) with one-line pointers to per-topic markdown files. Index loads every turn. Topic files load only when the agent actually needs them.
The whole thing is four tools:
read_memory_index()gives you the table of contentsread_memory_file(path)gives you one topic filewrite_memory_file(path, content)rewrites a file completelydelete_memory_file(path)deletes it
No embeddings. No chunker. No vector store. Just files, a naming convention, and a model smart enough to navigate them.
Why does it work? The index tells the agent what it knows. Topic files give depth on demand. And the size cap forces you to keep things short. Lots of small focused files beat one giant dump.
5. RAG Still Wins But Only in 4 Cases
This is the part "RAG is dead" posts skip. There are real cases where you need the full stack.
Huge unstructured piles. Tens of thousands of docs where the title doesn't tell you what's inside. Product manuals, legal archives, scientific papers, giant enterprise wikis. You can't list them all in an index. Exact-match lookups miss the relevant chunk. You need similarity search.
Regulated multi-tenant stuff. Healthcare, finance, defense. Vector stores give you row-level access controls and audit trails right out of the box. You can build this on a filesystem too, but you're building the primitives yourself.
Stuff that changes constantly. News feeds, market data, regulatory updates. Anything changing hourly. Vector indexes update incrementally. Filesystem memory drifts unless you build the same incremental path.
Agentic search over structured responses. When the agent reasons across thousands of records and decides what to ask next. That needs real retrieval with faceted metadata.
If your use case fits one of those four, build the RAG stack. If it doesn't, keep reading.
6. The Costs Nobody Scopes For
"A vector database" sounds like one line item. It's actually five systems you now own.
Chunking and embedding strategy. A support KB and a legal contract corpus should not be chunked the same way. Get this wrong and your retrieval caps out no matter which embedding model you pick.
A retrieval eval set. Separate from your task eval. You need to measure whether the right docs got retrieved, not just whether the final answer looked okay. Those are different measurements. Mixing them up hides real problems.
A freshness pipeline. Retrieval is only as good as the index is current. Point RAG at a KB that hasn't been reindexed in three months and it'll confidently cite outdated stuff.
Citations in the actual UI. If traceability is the reason you built this, retrieval that never reaches the user doesn't deliver anything.
Ongoing maintenance. Files go stale. Two sources contradict each other. The agent saves something wrong and repeats it forever.
None of this is exotic. But it's all real engineering work stacked on top of the retrieval call. That's why "RAG" needs its own line in the project scope not folded silently into "AI development."
7. The Decision Framework
Run your use case through four questions. If you say "no" to all four, skip the vector DB.
Does the answer depend on info that changes fast? Prices, inventory, ticket status, account history, this week's policy. If today's right answer differs from last month's, you need query-time retrieval.
Does the answer need to trace back to a source? In regulated or high-trust stuff, "here's the doc this came from" is often the actual product requirement.
Is the KB too big or too private to fit in context? If everything fits in a well-written system prompt with no per-user scoping, you need a better prompt. Not retrieval infra.
Is the data unstructured enough that exact-match fails? If SQL answers it, use SQL.
Any "yes"? Build retrieval. All "no"? Ship with prompting, long context, or file-based memory. Add RAG when a specific corpus actually demands it.
8. Start Simple. Add Retrieval When It Earns Its Spot.
My favorite stat on this comes from Anthropic's November 2025 post on code execution with MCP. A workflow burned ~150,000 tokens loading tool definitions upfront. They rebuilt it with filesystem-style APIs tool defs read on demand and it dropped to ~2,000 tokens. That's a 98.7% reduction.
They call it "progressive disclosure." Same idea as the memory pattern: load the index, load what you need, leave the rest on disk.
The sequence I'd recommend:
Ship v1 with prompting or long context. No vector store.
Watch how it breaks in production. Which questions actually need grounded, current, or private info?
Add retrieval when a real corpus demands it. Retrofitting RAG onto a working agent is a normal upgrade path, not a rebuild.
Same advice Anthropic gave back in December 2024, and it still holds: start simple, eval carefully, add complexity only when the simple stuff isn't enough.
The Short Version
A vector database is real engineering work. Not a checkbox. It adds an embedding pipeline, an index to keep fresh, an eval surface, and a second permission model sitting next to your DB.
Reach for one when your data is big, messy, changing fast, or needs per-tenant isolation and when a normal tool call genuinely can't answer the question.
For everything else, simpler wins. Your existing DB for structured data. Files and an index for agent memory. A long context window for static knowledge. You can always add retrieval later. You can't easily un-add it once it's in prod.
Best architecture is the one you can still explain six months from now and the one that fails in ways you can actually debug.
