distill your production traces into a cheaper model
the frontier api you call in production has been writing your training set this whole time. distill it into a model that keeps the quality and cuts inference cost 10x to 100x.
you built on a frontier api because it was the fastest route to quality, and it was the right call. the model handled your task well from day one, and you spent your time on the product instead of on training infrastructure.
a year in, the economics look different. every request pays for a model that can do almost anything, applied to the one thing your product needs. the bill grows linearly with traffic, and a multi-second round trip sits in your hot path.
the training set you already collected
every production trace is a demonstration of your task done well: a real input from a real user, and an output that was good enough to ship. months of them describe your task better than any synthetic dataset could, because they cover the distribution you actually serve, edge cases included.
distillation turns those demonstrations into a smaller model. we curate the traces (drop the failures, dedupe, balance across task types), then finetune an open model to reproduce the frontier behavior on your distribution. the student doesn’t need the teacher’s breadth. it needs your task, and the traces show it exactly what that looks like.
10x to 100x
that’s the cost range we see between a frontier api and a small open model you own, depending on how small the task lets you go. with elsa we post-trained a model on production chat traces from their english tutor; it performs better on the dimensions their product measures, at half the cost and latency. with neon, a 4b open model trained for retrieval over postgres data matched frontier accuracy at around one hundredth of the inference cost.
quality is measured, not assumed. a slice of your traces becomes a held-out eval, the distilled model runs against the frontier baseline on your own metric, and the run isn’t done until the gap is closed. when imitation alone doesn’t get there, the finetune becomes a warm start and reinforcement learning closes the rest.
what you end up with
a model that does your task at the quality bar you already trust, at a fraction of the price, wherever you want to run it. the weights are yours: no rate limits, no deprecation notices, no per-token pricing attached to your growth. and the loop continues, because every month of production traffic is new training data.
how it runs on castform
point us at your traces, whether that’s api logs or exports from your observability stack. we shape them into a training set and evals, run the finetune on managed gpus, and put the distilled model next to the frontier baseline so you can see the comparison before you move traffic. sft launches in under ten lines of code, and when a task turns out to need rl, the same traces seed the environment. you download the weights, or serve them on our inference.