The demo works.
An AI agent reads an inbound lead, researches the company, drafts a personal email, sends it, and queues a follow-up if nobody replies. The CTO shows it internally. People get excited. Budget gets discussed. Someone says, "Let's get this live in six to eight weeks."
Then the project slows down.
Usually the team is capable, the model did not suddenly get worse, and nobody faked the prototype. The prototype answered an easy question: can the model produce a useful output on a clean path? Production asks a much harder one: can the system do it again and again, safely, with memory, an audit trail, retries, and messy real-world input, when nobody is watching?
That gap is where many agent projects stall.
I know this pattern from both sides. When I worked on Eloqua at Oracle, generating output was never the hard part. The hard part was state, scheduling, reliability, bad inputs, and keeping operations correct at scale. At CoEdify we now build agent systems for real business workflows, and the same lesson keeps coming back: the model is one layer. The system around the model is the product.
Once an agent can call tools and wait on real-world events, prompt quality stops being the bottleneck. System design takes its place.
What the prototype hides
Here is a typical agent prototype, in this case a sales agent that handles new leads:
Agent receives lead -> researches lead -> drafts email -> sends email
You run it on 10 hand-picked leads. It works. The emails are good. The follow-ups land. You demo it, and everyone is impressed.
Here is what the prototype quietly assumes:
- The input is clean. In production, some leads have missing fields, wrong emails, stale data, or conflicting records.
- The tool calls behave. In production, APIs time out, hit rate limits, return partial results, or change without notice.
- The workflow is fresh. In production, the same person may already be in another campaign, another stage, or another thread.
- The conversation is linear. In production, threads fork. People reply late, forward emails, copy in colleagues, or answer on a different channel.
- The run is one-shot. In production, the system has to remember what happened yesterday and decide what to do next week.
- A human is watching. In production, failures surface after the fact, often as customer confusion rather than clean error messages.
The prototype handles none of this because it was never designed to. A prototype proves the happy path. Production is everything around the happy path, and that is where most of the engineering lives.
Six gaps that break agent systems in production
These failures come up again and again in the agent systems we build, whatever the use case.
1. No memory between runs
The prototype remembers nothing between runs. It handles each lead in isolation. In production, that does real damage.
A lead gets a draft on Monday. On Tuesday a different agent run picks up the same lead and drafts a completely different email. Two emails go out. The lead is confused, and you look careless.
Or worse: a lead replies "not interested." The agent never registers the reply because it has no memory, so it sends a follow-up. Then another. You have now spammed someone who already said no.
What production requires: a state layer that records what happened, when, and with what outcome. Before the agent acts, it reads the state. After it acts, it writes the state. Every interaction goes into an append-only log.
If you do not solve this first, every other improvement is fragile.
2. Bad output that nobody catches
In the prototype, the model always returns a well-formed email. In production, it sometimes returns:
- an email addressed to the wrong person
- a 2,000-word essay instead of a short note
- a message that invents a product feature you do not have
- an empty string
- a JSON object instead of plain text
The prototype does not check outputs. It assumes the model will be reasonable. Production cannot make that assumption. Every model output needs structured validation before it reaches a customer, a candidate, or another system.
from pydantic import BaseModel, field_validator
class AgentOutput(BaseModel):
subject: str
body: str
cta: str | None = None
@field_validator("body")
@classmethod
def body_must_be_reasonable(cls, v: str) -> str:
if len(v) < 20:
raise ValueError("Body too short or empty")
if len(v) > 500:
raise ValueError(f"Body exceeds 500 chars (got {len(v)})")
if "we offer" in v.lower():
raise ValueError("Output contains unverified product claim")
return v
@field_validator("subject")
@classmethod
def subject_must_exist(cls, v: str) -> str:
if not v or len(v) < 5:
raise ValueError("Subject missing or too short")
return v
Asking the model more politely will not fix this. Treat model output like any other untrusted external response: check it against a contract before it goes anywhere.
3. Dumping everything into the context window
The prototype sends the model a clean, hand-picked context: the lead's name, company, and a short prompt. The model handles it well.
Production is messier. The agent may need the interaction history, current stage, earlier replies, research notes, workflow rules, permissions, and brand rules. Teams pour all of it into the context window and call that "memory." It is a pile of text.
We saw exactly this in our own systems. Output was becoming generic for the most important contacts. The model was fine. The problem was how we assembled context: we sent too much and structured too little, so the model focused on nothing in particular.
A bigger context window does not fix this. The context window is temporary working space for one step; memory is state the system can recover later. Long prompts also have a known weakness: the Lost in the Middle study (Liu et al., 2023) found that models use information at the start and end of a long input better than information in the middle. So the useful question is not how much text the model can accept. It is how much this step needs to make its next decision well.
The fix is to keep memory outside the model and assemble context for each step. We keep three layers of memory in a separate store the agent reads from:
Layer 1: Research profile -> Written once, read often
Layer 2: Workflow context -> Changes often, kept compressed
Layer 3: Cross-workflow memory -> What we have learned over time
Before the agent acts, one call assembles the relevant slice from all three layers. It returns only what this step needs.
// Simplified: the shape of the call, not our production code.
const context = await memory.getContext({
tenantId,
contactId,
workflowId,
layers: ["profile", "workflow", "memory"],
});
// Returns a small, focused bundle for this step
// instead of a raw dump of everything we know.
A quick test: if the current stage or the last decision exists only because it is still sitting in the conversation history, the system is fragile. Move it into stored state.
4. Nobody knows when it breaks
When the prototype breaks, you see it, because you are watching. When production breaks, nobody notices until a customer complains, or until you find out the agent has been sending broken emails for days.
This is the observability gap. Most agent prototypes have no monitoring, no alerts, and no structured record of how the agent reached a decision. When something goes wrong, the first serious question is always the same: "What exactly did the agent do?" If you cannot answer that quickly, you do not yet have an operational system.
What production requires: a structured trail for every run.
Illustrative example:
[2026-04-14T02:31:07Z] INFO agent.run.start lead_id=L-4521 workflow=outbound_v2 stage=drafting
[2026-04-14T02:31:08Z] INFO context.read lead_id=L-4521 layers=[profile,workflow,memory] tokens=847
[2026-04-14T02:31:12Z] INFO model.call.complete model=anthropic/claude-sonnet-4-6 tokens_in=1247 tokens_out=312 latency_ms=4100
[2026-04-14T02:31:12Z] WARN output.validation lead_id=L-4521 result=rejected issue=body_exceeds_max_length chars=687
[2026-04-14T02:31:15Z] INFO model.call.complete model=anthropic/claude-sonnet-4-6 attempt=2 tokens_in=1302 tokens_out=164 latency_ms=3100
[2026-04-14T02:31:15Z] INFO output.validation lead_id=L-4521 result=passed chars=431
[2026-04-14T02:31:16Z] INFO state.write lead_id=L-4521 stage=drafted->queued_for_review
[2026-04-14T02:31:16Z] INFO agent.run.complete lead_id=L-4521 duration_ms=9100 status=success
Every run should leave a trail: when it started, what context it read, what the model returned, whether validation passed, what state changed, and how long it took. When something breaks, you reconstruct the run instead of guessing.
5. Runs that last for days
Prototypes are one-shot: you run them, they produce output, you stop. Production agents are long-running. They wait for replies, send follow-ups after delays, and retry on failure.
That means you need durable execution. If the server restarts, the agent must pick up where it left off rather than start the workflow again. If a timer says "follow up in 3 days," it has to survive deploys, crashes, and outages.
We started on Temporal for exactly this. On paper it was the obvious choice. In practice it did not fit how we run: a self-hosted cluster is heavy infrastructure for a lean platform, the managed cloud is another bill, and we still needed an agent-loop layer on top. Before launch we replaced it with Mastra, which has durable execution built in: suspend, resume, retries, and timers live in the workflow runtime instead of being buried inside agent code. We wrote the switch down as an architecture decision and moved on. Reversing an infrastructure choice before launch is cheap. After launch, it is a migration project.
The brand of engine matters less than this: the engine, not the agent, owns time and failure.
// Simplified from our execution layer (TypeScript + Mastra)
const waitForReview = createStep({
id: "wait-for-signal",
suspendSchema,
resumeSchema,
execute: async ({ suspend }) => {
// Park the run until a human approves - for up to 7 days.
// The run survives deploys, restarts, and crashes,
// then resumes exactly where it left off.
await suspend({ waiting_for: "signal" }, { timeout: 60 * 60 * 24 * 7 });
},
});
Our longest-lived runs park for days waiting for a human decision, then wake up mid-workflow with their full state intact. If the server crashes between steps, the workflow resumes from the last completed step. The lead does not get a duplicate draft or follow-up, and the agent does not lose its place.
One caveat we learned to respect: durable execution does not make side effects safe by itself. Every write still has to be idempotent, and every side effect still has to live inside a workflow step with a clear boundary. The runtime gives you the ability to resume. Safety is still your engineering job.
6. One customer's data reaching another
The prototype runs for one user: one set of leads, one brand voice, one workflow.
Production runs for every customer. Customer A's agent must never see Customer B's data, and Customer A's workflows must never change Customer B's pipeline. That is a hard requirement.
We enforce tenant isolation in the database schema as well as in application code. Every query is scoped to the tenant, and composite foreign keys stop a record from pointing at another tenant's data. Isolation then does not depend on every prompt and code path getting it right.
CREATE TABLE pipeline_stages (
tenant_id UUID NOT NULL,
stage_id UUID NOT NULL,
contact_id UUID NOT NULL,
stage_name VARCHAR(50),
entered_at TIMESTAMP,
PRIMARY KEY (tenant_id, stage_id),
FOREIGN KEY (tenant_id, contact_id) REFERENCES contacts(tenant_id, id)
);
You do not want to find a cross-tenant leak in production. Design it out up front.
The production readiness checklist
Before any agent system goes to production, we run through this checklist. Each item is there because we have already seen the failure it prevents.
- State management: Does the agent read state before acting and write state after acting?
- State outside the prompt: Do the current stage and past decisions live in stored state, not only in conversation history?
- Output validation: Does every model output pass a structured validator before it reaches a customer?
- Context assembly: Does each step get a focused slice of context instead of a dump of all data?
- Observability: Can you reconstruct any agent decision from structured logs?
- Durable execution: Does the agent survive server restarts without duplicating work?
- Shared records: Can two agents or workflows touch the same contact without conflicting or duplicating work?
- Multi-tenant isolation: Is tenant scoping enforced in the database as well as in application code?
- Error handling: Does the agent have defined behavior for API timeouts, empty responses, and malformed data?
- Human in the loop: Is there a review step before customer-facing actions?
- Rate limiting: Does the agent respect API rate limits and model token limits?
- Idempotency: If the workflow runs twice on the same input, does it avoid repeating external side effects?
If several of these are still open, you do not have a production system yet. You have a promising demo.
What the gap really costs
The biggest cost of the gap between demo and production is lost trust.
The team demoed something that worked. Leadership expected it to launch. When it does not, and month after month passes with "we are still working on edge cases," leadership stops believing the team can deliver. The AI initiative gets labeled experimental. Budget gets cut. Momentum dies.
The root mistake is building the prototype as if production were a deployment step. Production is an architectural requirement. It shapes how you design state, context, workflow execution, permissions, review steps, and observability from day one.
The work also pays off over time. Once the shared layer of state, context, and permissions is solid, each new agent becomes a thin layer on top of it instead of a new architecture project.
The teams that move faster in the long run are rarely the ones with the flashiest demo. They are the teams that treat the agent as one component inside a disciplined system.
The prototype proves the model can do the task once. The architecture proves the system can do it repeatedly, safely, and with accountability. That is the difference that matters.
At CoEdify, we build agent systems for real business workflows. State, context, validation, durable execution, and guardrails are part of the design from the start.