You built an assistant on top of your company's documents. In the demo it answered well. In daily use, people stop trusting it.

It quotes an old version of a policy. It answers a question about one customer with a general paragraph. It suggests a next step that ignores yesterday's call. The answers sound confident, and too many of them are wrong.

Teams usually respond by tuning the pipeline: new embeddings, different chunk sizes, a reranker, another vector database. The user experience barely moves, because the problem is in the design. The system was built to look things up in documents. It is being asked to act as working memory for a live business.

A demo question is clean:

"What does the documentation say about X?"

A real question is not:

"What do we know about this account, which policy applies, what changed last week, and what should we do next?"

At CoEdify we ran into this across several systems, and the pattern became clear quickly: plain RAG works for reference material and breaks once the agent also needs live context and history.

Where RAG works

RAG (retrieval-augmented generation: find relevant passages, then have the model answer from them) is useful when the question is about reference material:

  • product documentation
  • policy pages
  • process guides
  • specs
  • contracts

You have a body of fairly stable text. The user asks a question. The system retrieves the most relevant passages. The model answers from those passages. That is a valid pattern.

One place it works well for us is devsko, our AI hiring and assessment product. There, a job description can be used to retrieve relevant skill areas and role-specific question patterns for the assessment. That suits RAG, because the system is matching a job description against a curated reference layer of skills, question types, and role expectations.

The retrieval questions there are:

  • which skills matter for this role
  • which questions best test those skills
  • which assessment pattern fits this job family

That is reference retrieval. It is different from asking:

  • how this candidate performed three steps ago
  • what signals already appeared in the interview
  • what follow-up should happen next in this assessment

Those are questions about state and workflow, and RAG is the wrong tool for them.

The mistake is expecting the same retrieval pattern to also handle:

  • current workflow state
  • recent interactions
  • account-specific context
  • how a relationship has developed
  • decisions made outside the document collection

That is where disappointment begins.

Why production RAG feels unreliable

The embedding model or the vector database is rarely the real problem. The problem is that teams push different kinds of knowledge through one retrieval path and expect the model to sort it out.

In practice, the failures look like this.

1. It finds the right topic but the wrong version

A question about refund policy, pricing terms, or approval rules may return an older document because it is similar in meaning and richly worded.

The search sees similarity. It cannot tell which version is current unless you record freshness and status explicitly.

2. It returns general knowledge to a specific question

The user asks about one account, one workflow, or one open issue. The system returns a broad document, because that is what exists in the index.

The answer is on theme and still wrong in practice.

3. It misses the most important knowledge entirely

A lot of operational truth never makes it into documents. It lives in:

  • interaction logs
  • stage changes
  • call summaries
  • decisions captured in notes
  • approvals recorded in workflow state

If those facts are outside the retrieval design, the model answers from incomplete evidence.

4. It ignores what the system already learned

The last call changed the direction of the account. The latest approval changed the next action. The customer's pattern of objections is already visible from earlier conversations.

But the answer is generated from static chunks of text, without the latest state or any memory. That is why it sounds plausible and still feels wrong.

The core mistake: flattening knowledge

Most disappointing RAG systems share one habit: they put everything into a vector store.

That means:

  • documents
  • workflow state
  • interaction summaries
  • customer-specific context
  • accumulated relationship knowledge

all get treated as the same kind of retrieval object.

They are not the same. They change at different speeds, apply to different scopes, and need different ways of being fetched.

When you flatten them, the model gets text that is related in meaning but misses the operational context. The answer sounds informed and still misses the situation.

The fix: separate the knowledge layers

What worked for us was splitting the knowledge system into three layers and giving each one the right way to fetch it.

Layer 1: Reference knowledge

This is the classic RAG layer:

  • policies
  • documentation
  • product specs
  • standard agreements
  • internal playbooks

It changes fairly slowly and benefits from semantic search, metadata filters, and clear versioning. RAG belongs here.

Layer 2: Engagement context

This is the live state around a specific record or workflow:

  • current stage
  • recent interactions
  • latest notes
  • pending actions
  • context for this account only

This is a structured lookup, not a search problem. If the agent already knows which contact, deal, or ticket it is working on, it should fetch that record's state directly by its ID.

Layer 3: Accumulated memory

This is the compressed history that lets an agent act with continuity:

  • how this person usually responds
  • which objections keep coming up
  • which constraints matter in this relationship
  • what the system has learned across earlier interactions

Rebuilding this from raw history on every request is slow and inconsistent. Keep it as a short memory summary that is updated over time and cheap to read.

What this looks like in practice

Match the retrieval path to the kind of knowledge the agent needs:

  • For policy guidance, search the reference layer.
  • When devsko turns a job description into an assessment plan, it searches the reference layer.
  • For the current state of a deal or ticket, look it up in the engagement layer.
  • For continuity across earlier interactions, read the memory summary.

A simplified sketch:

def build_agent_context(question, contact_id, workflow_id):
    return {
        # Layer 1: semantic search over reference documents,
        # restricted to current versions
        "reference": search_reference_documents(question, current_only=True),
        # Layer 2: direct lookup of live state by ID, no similarity search
        "engagement": load_current_state(contact_id, workflow_id),
        # Layer 3: a short summary kept up to date over time
        "memory": load_memory_summary(contact_id),
    }

The names do not matter. What matters is that no single retrieval method pretends to solve all three problems.

devsko shows the boundary well. RAG can help identify which skills and question types are relevant to a job description. But once a candidate is in an active assessment, the system also needs structured state:

  • which stage the candidate is in
  • which answers were already given
  • which skills have already been tested
  • what follow-up is still missing

If all of that is flattened into one retrieval layer, the assessment becomes inconsistent for the same reason most production RAG systems do.

Why this matters to CTOs

If your team is building agents for support, sales, operations, hiring, or internal workflows, flat RAG puts a ceiling on reliability.

The model may sound fluent. The system will still fail when:

  • freshness matters
  • scope matters
  • state changes quickly
  • earlier interactions should shape the response

That is why many teams keep improving chunking, embeddings, and reranking while the user experience barely changes. They are tuning the wrong layer.

The bigger design question is:

What knowledge belongs in retrieval, what belongs in state, and what belongs in memory?

Once that is clear, the rest of the design gets much simpler.

A practical diagnostic

If your RAG pipeline keeps underperforming, check these in order.

1. Are you using RAG for a state problem?

If the question depends on live workflow state or recent interactions, start there. Vector search will not reliably recover system state.

2. Are old and current documents mixed together?

If yes, your retrieval layer needs explicit freshness and version controls.

3. Are account-specific facts mixed with general documents?

If yes, separate scoped context from reference knowledge.

4. Does the model have access to a compact memory?

If every answer has to rebuild history from raw records, the system will be expensive and inconsistent.

5. Are you evaluating retrieval, or only the final answer?

When an answer is weak, many teams blame the model first. Often the evidence it received was incomplete, stale, or about the wrong scope.

The takeaway

RAG is not broken. It is being asked to do too much.

Use it for what it is good at: finding reference knowledge in stable text. Do not make it carry live workflow state and relationship memory as well.

Separate those concerns and the system gets sharper quickly:

  • better answers
  • cleaner retrieval
  • smaller, less bloated prompts
  • more reliable agent behavior

That is the lesson we took from building our own memory layer for agents. The improvement did not come from a smarter embedding model. It came from treating reference retrieval, structured state, and memory as separate layers.

At CoEdify, we build knowledge systems for AI workflows in which RAG is one layer of the design. Retrieval, state, and memory are kept separate from the start.