Demos take a weekend now. Production still takes a fight.
Claude Managed Agents hit the scene saying their bots could run by themselves, no babysitting needed.
I called three founders who bought in. They launched: meeting briefs that check their own work, a dashboard tracking a whole squad of sales agents, and an agent that reads customer code and sends pull requests. All this, live in under two weeks.
Here’s the real story: the late nights, the tight deadlines, and the parts no one puts on a slide. This is what happens when you have to deliver for actual users.
Here’s how each one did it.
together with Oracle:
Bots are forgetful.
Oracle’s crew came up with a fix and spelled it out in a playbook:
They gave agents four kinds of memory: working, semantic, episodic, and procedural. In their tests, agents scored almost 94% on LongMemEval and used a tenth of the tokens after 80 turns.
▫️ Plays nice with any framework: Claude Agent SDK, LangGraph, OpenAI Agent SDK. Every one has a notebook to get you started.
▫️ One client and one pip install: oracleagentmemory.
▫️ Includes audit, tenant isolation, and ways to wipe data if you want.
Now, the founders. Same skeleton each time: what shipped, the clock, where it nearly broke.
1. One Wrong Name Nearly Shipped, and It Rebuilt the Product
Saj’s team builds a meeting assistant. Core feature: briefs. Who you’re meeting, what you agreed to last time.
Then one day it pulled the wrong LinkedIn profile for a real meeting.
“If you’re talking to a person named Tom and it gives you information about the wrong Tom, it’s actually almost destructive because the moment you end up in the meeting, you click to learn more about them and you end up confused.”
“Almost destructive” is the right phrase. A wrong name attached to genuine context beats an empty screen at one thing only: sending the user into a meeting armed with confident nonsense.
The fix: a second agent that grades every brief before a user sees it. Fail the grade, show nothing. Silence over confident error.
That’s the design principle behind outcomes, and I’d apply it anywhere your system touches someone’s identity.
2. A Sales Agent Fleet, Shipped in 2 Weeks (Not a Typo)
Mahir’s team went the opposite direction on scale. One agent per account, always running, each accumulating memory on its own account for as long as the relationship lasts.
Except reps rarely think in single accounts. A rep with 100 needs today’s 5. A manager needs next quarter. So a cross-account layer grew on top of the fleet: Watchtower.
“We spun that up on managed agents in about 2 weeks. It’s now one of our most used features, and it powers a lot of other parts of the product.”
Two weeks. I asked twice.
Build the narrow agent first. The cross-cutting need will introduce itself.
Planning to run the same math on your own product, start here:
Back to Saj’s grader, because the load-bearing detail is easy to miss.
“The fully independent clean context window means that you can assess the quality of the work as part of deploying it.”
The grader gets a rubric (right person, right context, readable structure) and its own context window. It never sees how the answer got made.
An agent reviewing its own trace will pass itself every time. Hand the grade to a stranger and the rubric starts meaning something.
4. Two Kinds of Memory, Learned the Expensive Way
Watchtower forced a distinction Mahir’s team now treats as gospel.
“There’s memory on an account and across accounts. There’s memory on a user level and an org level.”
Account memory is deep and specific: one customer’s history, tracked forever, tuned to stay stable. Cross-account memory is different: how this org defines a “qualified” deal, what its messy internal fields mean. Closer to infrastructure than to knowledge.
Mix the two and everything works fine for months. Then three engineers spend a quarter untangling it.
5. What Actually Needs a Sandbox
Todd’s product should make you a little nervous. It snapshots a customer’s codebase, compares it against the live application, and proposes fixes, often as a PR pushed straight to a live repo.
“You can imagine that people interacting with an agent, and you get access to someone’s source code, you can potentially get secrets.”
A code snapshot carries credentials, internal logic, things the customer never meant to hand anyone. Every tool that agent touches has to be scoped with that in mind from day one.
Todd’s team looked at building that isolation themselves and passed. Getting it right immediately mattered more than owning it. Unheroic, correct.
6. Build vs. Buy: 3 Founders, 1 Test
Three founders, three products. I expected three answers and got one.
“It really comes back to if you don’t need to roll your own infrastructure for it because it’s not the thing that defines the quality of what you’re building, choose to buy, and in the cases where it is, build it yourself.”
The test runs both directions. Saj: a voice-controlled assistant would need its own runtime, because there the runtime is the product. Briefs didn’t, so they bought that layer and spent the saved weeks on the feature. Todd came from the other side: homegrown solution, wall, proper bake-off against a managed option, then the rebuild decision.
Four questions before building anything yourself: control, cost, time, and (Todd’s addition) whether you’re still validating that the idea works at all.
Every founder at that table had rolled their own infrastructure once. Every one replaced it when it stopped being what made their product special.
7. Idea to 1,000x in a Few Weeks
The number that reframed the conversation for me: Saj’s briefs feature, self-grading included, took a day.
“It allowed us to build the first version of the feature in a day and then ramp up with like 100 to 1,000x scale up in the user base over the course of a few weeks.”
A day to working. A few weeks of dogfooding to 100-1,000x the users. Infrastructure untouched throughout. Watchtower ran a similar clock, with roughly one weekend of dogfooding wedged in the middle.
So the time went where it should: figuring out what users needed, testing against real usage, repeating. Once infrastructure is solved, speed is iteration cycles, never headcount.
8. The Eval Problem Nobody Has Solved
I pushed on evals. Every team admitted the same open wound.
“I don’t think we found a good way to eval given memory. A system that’s very stateful with this memory that is live.”
The early playbook is universal: trust your gut, dogfood internally, go systematic once it sticks. Dogfooding overfits to your own team; eval sets from real customer queries mostly cure that.
State is the part that stays broken. Live memory won’t hold still for a clean offline eval. Todd hits the same wall through MCP, where Slack tool calls carry state his team controls none of, so any eval suite drifts no matter how carefully built. Mocking the whole surface would work in theory. Nobody thought it was worth the engineering.
One founder gave me the industry’s actual state of the art in four words:
“Phase zero is vibes-based.”
If you’re building your own version of this loop, start here:
9. New Models, New Failure Modes (And Yes, More Em Dashes)
My favorite detour of the whole conversation: upgrades as a source of new problems.
“The whole five series of models tends to produce writing in a way that has actually a lot more AI telltale signs than previous series of models.”
The playbook these teams landed on inverts the upgrade instinct. Skip chasing the last 1% of performance, each generation hands you that for free. Spend the energy catching failure modes specific to the new model, whether that’s Claude Opus 5, Claude Fable 5, or Kimi K3, before a live user does.
A good eval suite catches most of it, on one condition: the prompts weren’t over-tuned to the previous model family’s quirks. Otherwise you’re testing nostalgia.
New model, new tells.
10. The Cost Levers, and the Missing One
Every founder here worries about cost. Good.
“If we’re running things 24 hours in advance, we could probably save 50% to 75% on cost and make that possible.”
That’s Saj describing a batch mode that fits briefs perfectly, since they run 24 hours ahead anyway, and would cut costs by more than half. It mostly doesn’t exist yet. Mahir already splits by task instead: the agent coordinating a Watchtower search gets frontier intelligence, the agents fanning across 500 accounts run cheap, because fanning out is cheap work.
Today’s levers: effort settings, orchestration that separates cheap work from expensive, model selection, and that missing batch mode, worth roughly 2x when it lands.
All three founders gave me a version of the same advice. Be mindful, spend well, and save the fine-grained optimization for when the product has earned it. Cost work pays off the day it makes something buildable that wasn’t. Before that day it’s a well-organized distraction from shipping.
Buy the stack when it’s a commodity. Build it when it’s the product. Every founder at this table proved both halves inside the same year, and it’s the test most teams skip before reaching for an agent at all.
▫️ Founders: 2 of the 3 products here started as one feature and became a second product only when a cross-cutting need showed up on its own. Ship narrow. Let the second product earn its place.
▫️ Investors: Teams that bought the stack spent their first 2 weeks studying user behavior. Teams that built spent it on infrastructure. Ask which one a team actually did.
▫️ Operators: A second agent with its own context window, grading against a rubric, catches what self-review sails past. Copy the outcomes pattern before you ship anything agent-graded.
▫️ Everyone else: Nobody at this table has closed the eval gap for live-memory systems. Assume your evals are stale until proven otherwise.
The 4 Principles to Steal
Buy until infrastructure is the product. Roll your own stack when it’s the thing that makes your feature different, and only then.
Grade with a second, independent agent. The agent that did the work loses its vote on whether the work was right.
Split memory by scope. Account memory needs deep customization. Cross-account memory is infrastructure. Mixing them is a bill that arrives later, with interest.
Hedge against new failure modes. The last 1% improves on its own with every release. New failure modes wait for you to find them.
Speed compounds. 2 of these teams shipped in 2 weeks and both are still on that layer today. The teams still rolling their own infrastructure are doing it on purpose, for the one piece that’s supposed to feel different from everyone else’s.
If this saved you a rebuild, send it to one founder or investor who needs it.
3 founders, 3 products, 1 conversation. Watch it start to finish.
Four kinds of memory just to stop an agent from forgetting its own context after 80 turns shows how much of "production AI" is really just memory management wearing a different name. Hit a smaller version of that building an AI shopping agent, forgetting what a user already said halfway through a session kills trust fast.
the grader point is the one i keep coming back to, but i'd add a caveat: an independent context window isn't an independent mind. if the grader is the same model family as the worker, they share blind spots and the grade is mostly theater for correlated failures. two different models, or a rubric with teeth — otherwise you're just paying twice for the same opinion.
Four kinds of memory just to stop an agent from forgetting its own context after 80 turns shows how much of "production AI" is really just memory management wearing a different name. Hit a smaller version of that building an AI shopping agent, forgetting what a user already said halfway through a session kills trust fast.
the grader point is the one i keep coming back to, but i'd add a caveat: an independent context window isn't an independent mind. if the grader is the same model family as the worker, they share blind spots and the grade is mostly theater for correlated failures. two different models, or a rubric with teeth — otherwise you're just paying twice for the same opinion.