optimize agent responses to improve customer outcomes
you already record what happened after each conversation. train the agent against that record, not against what sounds good.
two support conversations read equally well. same tone, same structure, both factually right. one customer renewed the following week, the other churned. a frontier model can’t tell them apart, because nothing in its training ever connected a response to what the customer did next.
prompting inherits that blindness. ask for “concise, empathetic, proactive” and you’ll get responses that sound like all three. but sounding right is a proxy. the numbers your business runs on live in another table entirely: retention, conversion, resolution, engagement.
behavior plus outcome is a training signal
if you log your agent’s conversations and you record outcomes, you’re already holding both halves of a reinforcement learning problem: what the agent did, and what it led to. joined together, every conversation becomes a labeled example of what actually works on your customers, at the scale of your production traffic.
how the training works
first we train a reward model on those pairs. its job is to predict the outcome from the conversation: which clarifying questions precede resolved tickets, when a discount saves the account and when it just spends margin, what pacing keeps a user coming back tomorrow.
then we build an environment out of the same traces, so the agent has someone to practice on: a simulated user that phrases things the way your users phrase them, pushes back where they push back, and loses patience the way they do. it’s grounded in recorded behavior, not an invented persona.
then reinforcement learning. the agent plays out thousands of conversations in that environment, and the reward model scores each one by predicted outcome. behaviors that move your metric get reinforced. the model that comes out is optimized for your number, which is the part no frontier api can offer: those models are tuned to a generic rater’s approval, and your retention curve was never in their training data.
this is how we trained elsa’s tutor
elsa, an english learning app, brought production conversations from their ai tutor. we built rewards from what their team knew mattered to learners, spent real effort on anti-reward-hacking machinery (the model discovered emoji spam long before it discovered pedagogy), and post-trained a model that performs better on the dimensions their product measures, at half the cost and latency. the write-up covers the reward design in detail, including the ranking setup that made the judges hard to fool.
how it runs on castform
bring conversation logs and an outcome column: renewed or churned, converted or bounced, resolved or reopened. we join them, train the reward model, construct the environment, and run rl on managed infrastructure. the lift is measured against your current agent on held-out conversations before anything ships. and because the weights are yours, next quarter’s outcomes can train next quarter’s model.