Why watching your agent think changes everything

1. You can’t staple an agent onto a request.

Here’s the thing nobody warns you about. Your first instinct will be: “Agents are slow, so I’ll just handle them like any other API endpoint. Spawn more workers. Done.”

Try it. You’ll notice something weird. Your p99 latency on unrelated endpoints, like GET /health or POST /settings, starts drifting up. You’ll check your CPU. You’ll check your database. You’ll check your network. Everything looks fine. Except your uvicorn workers are stuck 80% of the time on agent requests, and every other endpoint is queueing behind them.

The subtler issue is deploys. Every time you push a new version of your API, the orchestrator rolls your pods. That kills every in-flight agent run. Users see a broken conversation mid-sentence. You can’t ship without coordinating with the agent clock, which you don’t want to do.

The fix is to separate the two things. HTTP requests are short, so keep those in the API. Agent runs are long and stateful, so move those to their own process with their own lifecycle. Now your API can roll at will. The worker can recover from crashes on its own schedule. You’ve stopped stapling a marathon onto a sprint.

This isn’t about “using a queue.” It’s about noticing that two things you thought were the same thing are actually different things.

2. The best message bus is the one your team already runs.

You’ll read a dozen articles about Kafka and come away thinking that’s what the big companies use, so maybe you should too. Fine. Go deploy a 3-node Kafka cluster. Set up KRaft. Configure partitioning. Write a consumer group. Come back and tell me what your weekend was like.

Now look at the actual shape of your problem. One producer per agent. A handful of consumers per conversation. Events that live for maybe a minute. Payloads of a few kilobytes of JSON. Kafka was built for the shape of LinkedIn’s activity feed: massive throughput, many producers, many consumers, durable forever. Your problem is a different problem. You’re paying enterprise operational cost for kindergarten-scale traffic.

Redis Streams is what you actually want. It’s append-only. XRANGE catches you up from any point, which is reconnection built in. XREAD BLOCK wakes up clients when new events arrive. Latency in microseconds. None of it is free forever, but all of it is free right now.

And (this is the part most comparisons skip) Redis is probably already in your stack. You’re using it for sessions, for rate limits, for cache. Adding streams is one more command on a connection you already have. No new deploy, no new ops runbook, no new thing that might go down at 3am.

If you don’t have Redis yet and you’re still small, don’t add it yet. A Postgres table with LISTEN/NOTIFY does the same job up to a few hundred concurrent streams. It won’t scale forever, but by the time it doesn’t, you’ll know.

The general move: match the tool to the shape of your problem and the shape of your team. Then stop.

3. Two different guarantees need two different writes.

Here’s a thing that took me a while to get right.

You’ve built your worker. It’s producing events. You need to get them to connected clients and to clients that briefly disconnected. Both matter. If you pick Streams only, connected clients work fine (they XREAD BLOCK 0 and get notified), but each live client holds a Redis connection open, and your connection pool grows with your user count. If you pick Pub/Sub only, fan-out is cheap, but there’s no way to recover events a client missed during a 2-second wifi blip. They’re published, nobody was listening, they’re gone.

The obvious-in-retrospect move is to write to both. XADD for durability, so the event is there, replayable, until you delete it. PUBLISH for immediacy, so here it is, right now, to anyone listening. One event, two writes.

It costs one extra Redis command per event. In exchange, your reconnection path becomes trivial: client reconnects, sends Last-Event-ID: 42, server XRANGEs from 43 forward, then subscribes for live. No gaps, no special handling, no “reconnection logic” in the application layer.

There’s a subtle invariant worth naming. If XADD succeeds and PUBLISH fails (say, a Redis hiccup between the two commands), the event is durable but live subscribers miss it. That’s not a bug. Connected clients will see it when they reconnect and replay. The contract is durable, always; live, usually. That’s enough.

The principle: durability and liveness are two different properties of your system, and they don’t compose from a single primitive. When you need both, stop trying to be clever with one. Use two things for two jobs.

4. The transport is not the record.

Redis is fast. It’s in memory. It has streams, pub/sub, low latency, cheap ops. You might be tempted to just… use it for everything. Let the Redis Stream be the permanent record of the conversation. Never delete it.

Don’t.

Here’s what happens. Your Redis hits its maxmemory limit, which it will, because you never expire anything. Eviction kicks in. By default, Redis picks the least-recently-used key to evict. The “least-recently-used key” is going to be somebody’s active conversation, the one they just walked away from for coffee, the one they’re about to come back to. That user sees: “Your conversation disappeared.”

This is the kind of failure that’s invisible in testing because your test conversations all fit in memory. It’s catastrophic in production once memory is saturated. And by the time you discover it, Redis is already in pain and you have no good options.

The fix isn’t “tune the eviction policy.” The fix is to notice what Redis is actually for. It’s transport: the fastest way to get an event from a worker to a browser. It is not the historical record of what the user’s AI said yesterday. That record belongs in Postgres, or any real database. Cheap to store. Predictable under memory pressure. Easy to query.

Once you split the two, everything gets simpler. Events flow through Redis during a turn (XADD for durability, PUBLISH for live delivery), and each one also lands in Postgres at write time. When the turn completes, you delete the Redis stream. Its job is done; the Postgres row persists. Cleanup is trivial: delete at completion, set a short marker key so late-arriving retries don’t resurrect the stream. The marker is a little ugly, I agree. It’s ugly in a bounded way.

The principle generalizes. Every piece of infrastructure in your system has a job. Caches cache. Queues queue. Databases persist. You will be tempted, regularly, to use one of them for something else because it’s fast or already there. Resist. The systems that scale are the systems that let each component do one thing well.