AI That Survives Its Second Month in Production.
Most AI work fails after the demo, not during it. We build the unglamorous half — evaluation, monitoring, fallbacks and the human review path — so the thing still works when the data shifts.
The state of play when we arrive.
- A pilot that impressed everyone and never reached production.
- No way to tell whether the model got better or worse after the last change.
- Accuracy quoted on a curated sample that nothing in production resembles.
- No answer for what happens when the model is wrong — because it will be.
What we do about it.
- An evaluation set before a model: real inputs, labelled, held back, and version-controlled.
- The smallest thing that clears the bar — rules or classical ML before an LLM, if that is what the data wants.
- A human-in-the-loop path for low-confidence cases, designed as part of the workflow rather than bolted on.
- Monitoring on drift, latency and cost per call, so degradation surfaces before a user reports it.
Five phases, and what each one leaves behind.
- 1 Requirement analysis What decision does this change, and what is the cost of being wrong? If neither has an answer, we say so before you spend.
- 2 Data collection & preprocessing Sourcing, labelling and a held-back evaluation set. Usually the longest phase, and the one that decides the outcome.
- 3 Model development & training Baseline first, then earn every increase in complexity against the evaluation set.
- 4 Integration & deployment Behind an API, in your workflow, with confidence thresholds and the review queue wired in.
- 5 Optimization & support Drift and cost monitoring, scheduled re-evaluation, and a retraining trigger that is a rule rather than a hunch.
Where the work lands.
Before you book the call.
Do we need our own data?
For anything predictive, yes — and enough of it, labelled. For retrieval-grounded assistants your documents are usually sufficient. Phase one is where we tell you honestly which of those you are in.
Which models do you use?
Whichever clears the evaluation bar at acceptable cost and latency. That is often a hosted frontier model, sometimes an open-weight model you run yourself when data residency requires it, and sometimes no model at all.
How do you handle data residency?
It is a design input, not an afterthought. Where data cannot leave a jurisdiction or a VPC, that constrains the model choice, and we scope against it from day one.
What if accuracy is not good enough?
Then we say so and stop. An honest "the data will not support this" in week three is worth more than a system nobody trusts in month six.
Want this looked at properly?
Book a 30-minute discovery call. We'll tell you what we would do first — and whether we are the right people to do it.