Fine-tuning or a better prompt? When a custom model genuinely pays off

“We would like our own model, trained on our data.” We hear this regularly, and in most cases my first answer is “you probably do not need one”. Not because it cannot be done — I have been training models for years. But because training is the most expensive route to a result that can often be reached more cheaply.

Here is the decision process we use.

Four levels, in this order

You work upwards. You move to the next level only when the previous one demonstrably falls short.

1. A better prompt and structured output

It sounds trivial, but a surprising number of problems end here. If the model gets precise instructions, examples of the desired output and a schema to answer into, accuracy often rises by tens of percent.

Cost: hours of work. When it falls short: when the model has no way of knowing the answer.

2. RAG — supply the context

If the problem is that the model does not know your products, policies or customer history, the solution is not training but putting the information into the context. There is no need to teach a model facts that can be looked up.

This is most often the correct answer to “train it on our data”. In nine cases out of ten, a company does not want a model with the data inside it — it wants a system that can reach the data.

Cost: weeks. When it falls short: when the problem is not facts but style, format or domain reasoning.

3. Fine-tuning

Only here does training start to make sense. Fine-tuning is good for teaching a model how to answer, not what:

• a consistent output format that prompting cannot reliably enforce,
• domain terminology and the way reasoning works in your field,
• shrinking the model — a smaller fine-tuned model often handles a narrow task better and cheaper than a large general one.

That last point is often overlooked and is economically the most interesting. If you process hundreds of thousands of requests a month, the per-token price difference between a large and a small model repays the training cost very quickly.

Cost: low single-digit thousands of euros for a language model, if you have the data. When it falls short: when the task is not textual, or when you need something the base model cannot do at all.

4. Training your own model

In practice this mainly concerns computer vision and specialised tasks: quality control on a line, detection in satellite or drone imagery, measurement from images. Here there is often no base model that can do your task — it has to be trained.

Cost: tens of thousands of euros, mostly for annotation. When it makes sense: when the task is the core of your business, not an add-on.

Four situations where a custom model genuinely makes sense

1. Data must not leave the company. Legislation, a customer contract or an internal rule. This is the most common legitimate reason and it is entirely valid.
2. A high volume of similar requests. Above a certain volume a smaller custom model pays for itself on saved tokens alone.
3. Low latency or offline operation. A production line cannot wait for a cloud response and cannot stop when the network drops.
4. A narrow domain where general models fail. Specialised technical texts, or image data the model has never seen.

If none of these apply, start without a custom model. You can always add one later — and by then you will have data and a test set from production, which makes training considerably easier.

How much data do you need

The most common question, and the answer is uncomfortably vague because it depends on the task.

• Fine-tuning a language model for format and style: often 200–1,000 good examples are enough. Quality matters more than quantity — a hundred consistent examples beat a thousand contradictory ones.
• Text classification: on the order of hundreds to thousands of examples per class.
• Computer vision: on the order of thousands of images, and above all coverage of variability — different lighting, angles and failure states.

More important than the count is annotation consistency. If two people label the same thing differently, the model learns that contradiction. Checking inter-annotator agreement is a step worth taking before training, not after.

What to demand before agreeing to training

Three things without which a training project should not start:

A baseline. What accuracy do you get with an existing model and a good prompt? Without that number you cannot tell whether training achieved anything.

A held-out test set. Data the model did not see during training. Measuring on training data is worthless, and unfortunately it happens.

A reproducible pipeline. In a year a better base model will arrive. If you can repeat the training with one command, the switch is cheap. If not, you start over.

Summary

The order is prompt → RAG → fine-tuning → custom model, and most companies stop at the second level. That is not a defeat, that is a saving.

If you want to know which level your problem belongs to, custom AI models starts precisely with a feasibility study — and its conclusion is surprisingly often a recommendation not to train.