From minutes to under a minute: a guest messaging RAG rebuild

Challenge

The guest was waiting minutes for an answer, and the retrieval layer underneath the agent was the reason.

Cortado’s product answers guest messages with LLM agents. Answering one well is a retrieval problem before it is a language problem: the agent has to have the right facts about the specific property in front of it, and those facts sat across separate sources rather than in one place.

The system built to pull them together had grown past the job. The CTO’s own description is an over-engineered behemoth, and over-engineering shows up as latency, because every extra hop between the question and the facts is time a guest spends looking at a typing indicator. Responses were taking minutes. In a rental market where a guest can ask the same question of somebody else, that is not a technical detail.

The ingestion side had the same shape from the other direction: one monolithic pipeline, where every stage moved at the speed of the whole.

Approach

We rebuilt retrieval from first principles instead of optimizing what was there.

That distinction did the most work in this engagement. Optimizing an over-engineered system tends to preserve the thing making it slow, because the parts that cost the most time are usually the parts hardest to remove without deciding they were never needed. Starting from what an agent actually requires at answer time makes that decision explicit rather than inherited.

Two choices followed from it. Retrieval gets one place to read from, so an answer no longer depends on which source happened to be consulted. And ingestion stops being a single unit, so a slow stage stops setting the pace for the rest.

Solution

One retrieval path, one store behind it, and an ingestion pipeline that runs in pieces.

Property data consolidated into a Pinecone vector database. The facts an agent needs had been spread across separate sources. Bringing them into one vector store gave the agents a single place to retrieve from, which is what improved response accuracy: relevance became a property of the retrieval, not of which source the question happened to reach.

A leaner retrieval system, rebuilt rather than tuned. The rebuilt path does what the agent needs at answer time and not more, which is where the latency came back.

Ingestion moved from a monolith to distributed, event-based microservices. Stages publish and consume events instead of running as one block, so processing latency dropped and a stage can be changed without the pipeline being redeployed around it.

Results

Agent response latency dropped from minutes per response to under a minute.

Retrieval accuracy improved and customer satisfaction improved. Neither of those came with a figure from the client, so neither gets one here. That is the whole reason the metrics strip above carries a latency number and a count and nothing else: the improvements the client reported qualitatively stay qualitative, and inventing a percentage for either would be the easiest lie on this page to tell and the hardest to notice.

What the engagement does evidence is the shape of the fix. A retrieval layer that had accumulated more machinery than the problem needed was replaced by one that did not, and the response time moved by an order of magnitude.

What made it hard

Deciding what to delete is harder than deciding what to build. An over-engineered system is rarely built carelessly; it is usually built by people responding to real constraints one at a time. Cutting it down means working out which of those constraints were real and which had been designed around long after they stopped mattering, and being wrong in that direction removes something load-bearing.

Consolidating separate sources forces the question of what the canonical answer is. Once several sources become one store, anywhere they disagreed previously resolved itself by accident, according to which source a query reached. After consolidation that resolution has to be a decision somebody made on purpose.

The ingestion migration ran under a live product. A monolith-to-events move is easy to describe and unforgiving to sequence, because the old path has to keep serving guests while the new one takes over stage by stage, and the intermediate states are the ones nobody designs for.

One honest limitation, for anyone reading this to judge whether it matches their problem. The instrumented outcome here is latency. Accuracy and satisfaction were reported as better by the people who own the product and were not measured into a number that could be quoted, which is a normal state for a retrieval rebuild and still a gap worth naming.