Four rungs · Cheapest first · Harness included
Fine-tuning, retrieval, evaluation, and custom models — with an honest answer about which one you actually need.
Definition
Adapting AI models to a specific task or domain: fine-tuning a foundation model on curated examples, building retrieval systems that ground responses in your own data, designing evaluation harnesses that measure whether output is actually correct, and training purpose-built models for narrow prediction tasks where general-purpose models underperform.
Most teams who come to us asking for a custom model don't need one. We'll tell you that before you spend anything.
The escalation ladder
There are four ways to make a model behave the way you need. They're ordered by cost and complexity. The discipline is to climb only as far as you have to, because every rung adds maintenance burden that never goes away.
Structuring the instruction, the examples, and the context window properly. Unglamorous, and it resolves more problems than anyone expects — a large share of “the model isn't good enough” turns out to be “the model was asked badly.” Costs almost nothing to try and adds no maintenance surface.
Stop here if: output quality improves acceptably with better instructions and a few well-chosen examples.
Indexing your documents, policies, records, or knowledge base and retrieving the relevant pieces at query time so the model answers from your material rather than its training data. This is the correct answer to the vast majority of “we want it trained on our content” requests. It updates when your content updates, it cites its sources, and it doesn't require retraining anything.
Stop here if: the problem is the model not knowing your facts. This is almost always the rung people actually needed.
Adjusting a foundation model's weights on curated examples so it reliably produces a particular structure, tone, format, or classification behavior. Effective for consistency and for domain-specific output conventions. Poor at reliably installing new factual knowledge, and it has to be redone when you move to a newer base model.
Climb here if: you need consistent output shape that prompting can't hold, or a specialized classification behavior — and you have curated examples in the hundreds, not dozens.
Training a purpose-built model on your proprietary labeled data — a classifier, a forecasting model, a scoring system. Genuinely right for narrow, well-defined prediction tasks where you have real labeled history and a general-purpose language model is the wrong tool entirely. Not the right answer for anything conversational.
Climb here if: the task is narrow prediction on structured proprietary data, you have labeled history, and accuracy requirements are specific and measurable.
Most engagements we run end on rung two. Some reach rung three. Rung four is real, and it's rare, and we'd rather tell you that now than three months into a contract.
The correction
This is the most common and most expensive misunderstanding in applied AI, and nearly every team arrives with it backwards. “Train it on our data” almost always describes a retrieval problem, not a fine-tuning one.
The two combine well. A fine-tuned model that formats output correctly, grounded by retrieval that supplies the facts, is a common and effective architecture. What doesn't work is fine-tuning as a substitute for retrieval — you get a model that has absorbed the flavor of your documents and will confidently invent details from them.
Where engagements land
02
The rung most teams actually needed
Teams arrive asking for rung three or four. The failure they're describing almost always lives on rung one or two — a badly structured prompt, or a model that was never given access to the facts it's being asked about.
Climbing costs money twice: once to build, and then permanently, in maintenance. So we diagnose first, and we say so when the cheaper rung is the right one.
Evaluation
Ask any AI vendor how they measure whether their system is correct. The quality of that answer tells you almost everything. Here's ours.
A set of representative inputs with known-correct outputs, assembled with your subject-matter experts. This is tedious and it is the difference between engineering and guessing. Without it, “it seems better” is the only available verdict.
Every modification — prompt, retrieval strategy, model version, tuning run — gets scored against the golden set. You see whether a change helped, hurt, or moved things sideways, before it reaches users rather than after.
Fixing one failure mode routinely introduces another. A regression suite catches that. It's standard practice in software and startlingly rare in AI projects.
Providers update models, deprecate versions, and adjust behavior on their own schedule. Your evaluation harness is what tells you the day their update broke your application, instead of your users telling you a week later.
Scenarios
Three shapes of problem we see repeatedly, and where each lands on the ladder.
An admissions or member-services team answering the same forty questions all year. The instinct is to train a model on the handbook. The right build is retrieval over the handbook plus a clean escalation path to a human — because policies change every cycle and a retrained model goes stale while an indexed one doesn't.
Applications, referrals, or case notes arriving as free text that a person has to read and categorize. This is a genuine fine-tuning case: the categories are yours, consistency matters more than eloquence, and you likely have years of historical examples already labeled by the humans who've been doing it.
Flagging unusual transactions, predicting which donors lapse, forecasting demand. Structured data, labeled history, a measurable accuracy bar. A purpose-built model beats a language model here decisively, and costs far less to run at volume.
Deliverables
Including the evaluation harness — the deliverable most vendors quietly keep.
Process
Short cycles with real numbers, including the cycles where a change didn't help.
We establish what's actually failing and where on the ladder the fix lives. Frequently this conversation ends with a smaller engagement than the one you came in asking about. That's the outcome working correctly.
Before any implementation, we assemble the golden dataset with your experts and establish a baseline score for the current state. Without a baseline there's no way to prove improvement, only to assert it.
Build, score against the golden set, review, adjust. You see the numbers each cycle, including the cycles where a change didn't help. Especially those.
You receive the evaluation suite, the datasets, the training and retrieval configuration, and documentation. The harness is what lets your team keep improving the system after we're gone, and it's the deliverable most vendors quietly keep.
Qualification
Related
When the model is one step inside a larger workflow and the real work is orchestration and deciding where a human stays in the loop.
ExploreGetting a model into production and keeping it healthy there — versioning, drift detection, and the running-cost management nobody budgets for.
ExploreThe parent engagement — model work plus the interface and integration that turn it into something people use.
ExploreQuestions
Send us the failure you're seeing. We'll tell you where the fix lives — even when that's a smaller engagement than the one you came in asking for.
Fine-tuning teaches form. Retrieval teaches facts.
Book a consultation