Client-reported: 100 questions to 33 consultations in 2.5 weeks

Challenge

The firm’s website took questions and did nothing with them. Somebody with a real problem would land on it at eleven at night, find a contact form, and either fill it in and wait or leave. The ones who left were the ones the firm never heard about.

The obvious fix is an assistant that answers the question, and the answer itself is what stops most firms shipping one. Telling a member of the public that they qualify for something, or that their matter will succeed, is the practice of law. A system that does it once, in any language, is a liability rather than a source of work. Everything in the build follows from that.

Approach

We separated the two things a good answer does at once: explain what the options are, and decide which one applies. Explaining is information, and software can do it. Deciding is legal judgment, it belongs to the attorney, and we built the system so it cannot drift into that seat.

The attorney wrote the behaviour rules. We built the system that enforces and verifies them, rather than writing a prompt and hoping. The pipeline applies those rules as checks instead of asking the model to remember them, and a separate test tier grades every change against them. That is what let him put his name on it.

Solution

A single-shot question, answered in public. A visitor asks one question, with no account, in their own language. A cheap model screens it, an expensive model writes the answer with live web search for current source material, and a one to three page overview arrives in the browser in 20 to 90 seconds. It closes with a consultation offer routed to the firm’s calendar.

A compliance layer enforced in the system. The response never determines eligibility and never predicts an outcome. It names any complication the visitor discloses up front and in plain terms, rather than softening it into more comfortable phrasing. The system admits it when a question does not carry enough to answer, and routes to a consultation instead of assuming facts and building on them. Every response closes with the firm’s disclaimer, translated when the answer is not in English.

A fourth test tier that grades the rules, not the code. Unit, integration and end-to-end tests say the pipeline works. None of them say the answer was safe. So each scenario pairs a fixture with an ideal response the attorney wrote himself: deterministic checks cover the rules he made non-negotiable, and an LLM judge scores the rest against his version. It runs outside the per-commit suite, so the tests that run on every push spend no model budget, and the pipeline logs each run against the prompt that produced it.

A curation loop that captures the attorney’s judgment. He reviews candidate answers in an internal review application, rates them and annotates them. The system embeds the ones he selects, retrieves them against the live visitor’s question, and injects them as examples. The model is never retrained. The selected answers feed retrieval instead, which keeps his judgment readable rather than baked into weights nobody can inspect. The client drew that line before the work started.

Results

Client-reported, from the first 2.5 weeks after launch at a practice we do not name. The figures are the client’s own and approximate.

• Roughly 100 visitor questions answered, of which about 33 became booked consultations.
• About 15 of those consultations were attended, and about 5 became signed matters at roughly $5,000 each.
• A question-to-booking rate near one in three, which is the number the platform exists to move.

Two honest notes. These are the client’s figures rather than our own instrumentation, and 2.5 weeks is a short window, so a rate measured over a longer one would probably look different. And the gap between booked and attended is real: about half the people who booked a consultation did not attend it. That is a scheduling problem the assistant does not solve, and we are not going to claim it does.

What made it hard

Proving the rules held was harder than writing them. A rule like “never predict an outcome” is easy to put in a prompt and impossible to verify by reading code. The fourth test tier is what closed that gap. Without it every prompt change is an unmeasured risk to the one property that makes the system publishable, and the failure surfaces in front of a member of the public rather than in a test run.

The second problem was time. A generation that can take a minute and a half cannot hold a connection open and cannot block a request. The pipeline claims each job atomically, which stops two workers taking the same one, pushes the finished answer to the browser instead of making it wait on a socket, and sweeps for jobs that were claimed and never finished. A visitor who closes the tab and comes back still gets an answer.