AI Agents and LLM Systems
An AI agent or LLM system is software that reads your documents and answers questions about them, so the reading and sorting your team does by hand becomes a queue they review instead. It earns its cost when the work is high volume, repetitive, and made of text: filings, contracts, tickets, emails, and the questions staff ask the same manager every day.
Where it fits
The work has to be worth automating. A process that runs twice a year is a bad candidate no matter how tedious it is. The ones that pay off are the ones that run daily, involve reading something, and end in a decision a person makes the same way each time.
Three shapes come up most often. Document extraction turns a stack of unstructured files into records you can query, which is what makes reporting possible at all. Retrieval and question answering puts an assistant in front of a body of knowledge so people stop interrupting whoever holds it. Routing and triage classifies what arrives, handles the routine cases, and escalates the rest.
What makes them work
Accuracy is a number, not an adjective, and it has to be agreed before the build. On one extraction pipeline the acceptance criterion was 75% or better, and a two-week validation window came in at 80% or better. Separately, 1,664 filings ran through classification and extraction at a 98.8% success rate. Those figures are only meaningful because the bar existed first.
Traceability is the second requirement. Every answer points at the document it came from. That is what makes the output reviewable, and review is what makes the system safe to rely on.
The technology is supporting detail
These systems are built with Python, self hosted orchestration, retrieval over a vector store, and whichever language model fits each step. Which model is a design decision that gets revisited, not a pitch. What matters is the accuracy bar, the traceability, and whether the thing removes work.
The problem
Your team reads, sorts, and answers the same documents and questions every day, and the volume grows faster than the headcount you can add.
The outcome
A system that reads the documents, answers the questions, and routes what needs a person, with every answer traceable to the source it came from.
What you get
- A retrieval layer over your documents, so every answer cites the source it came from
- Extraction pipelines that turn filings, contracts, and forms into structured records you can query
- Assistants that answer staff or customer questions in Slack, WhatsApp, or whichever channel your team already has open
- An evaluation harness that measures accuracy against an acceptance bar agreed before the build starts
- Model routing, so each step runs on the model that fits it rather than the most expensive one
- A human review queue for the cases the system should not decide on its own
How the work runs
Set the acceptance bar
Build retrieval and extraction
Measure against the bar
Ship with a person in the loop
Related work
eight
specialized assistants reachable from one Slack conversation
Eight AI assistants a creative agency talks to in Slack
Crescendo Creative
roughly $77M
political TV ad spend tracked and attributed
Turning FCC filings into roughly $77M of tracked political ad spend
a US political advisory practice
an estimated 75%
fewer manager interruptions, by the client's own estimate
An estimated 75% fewer manager interruptions across 5 locations
Padrino's Cuban Cuisine
under 2 minutes
to compile the weekly lead report, down from 5 hours
From 5 hours to under 2 minutes: a legal firm's lead report
a US legal services firm
Questions about this service
How accurate are these systems in practice?
Accurate enough to be measured against a number you set in advance, which is the part that matters. On a document extraction pipeline we built, the acceptance criterion agreed before the build was 75% or better, and a two-week validation window came in at 80% or better. Separately, 1,664 filings ran through classification and extraction at a 98.8% success rate. Your documents will produce different numbers, so we set the bar with you first and report against it.
What stops it from making things up?
Retrieval and traceability, not a promise. Answers are generated from documents the system retrieved, and each one carries a link back to the source it used, so a wrong answer is something you can find and correct rather than something you discover later. Where the cost of being wrong is high, the system routes to a person instead of answering.
Do we have to move our data somewhere new?
Usually not. These systems read from where your documents already live and write structured results back to a store you control. Where a new store is genuinely the right call, we say so and explain what it buys you.
Which model do you use?
Whichever one fits the step, and that changes over time. Classification, extraction, and conversation have different accuracy and cost profiles, so routing each step to an appropriate model is part of the design rather than a single vendor decision. The system is built so a model can be swapped without a rewrite.
Not sure which of these you need?
That is the usual starting point, and it is what the assessment is for. Tell us which process is costing you the most time and we will tell you whether it is worth automating.