LOOP
Building a Production AI Agent in Spring Boot: Canary Releases, Model Fallback, and Cost Caps (Part 10)
**Production AI Agent in Spring Boot: Canary Releases, Model Fallback, and Cost Caps (Part 10)**
Last Monday I shipped the shipping-tool prompt from Part 9 to 5% of live traffic. It had won the pairwise gate 24 to 7 with 9 ties, the tool discipline diff was clean, and the money-path cases read fine by hand. I went for lunch confident.
At 6:42pm the canary dashboard disagreed. P95 latency was up 38%. Tool calls per conversation had climbed from 3.1 to 5.4. The reworded tool description I was so proud of had taught the agent to call the shipping tool twice per turn, and in real conversations, which run much longer than my 40 test cases, every extra call doubled the wait. I rolled the split back to zero in nine minutes, and the numbers returned to baseline by 7:15.
The 40-case dataset from Part 8 could not have caught this. Its transcripts are short by design. The pairwise judge from Part 9 could not have caught it either, because it judges two responses, not the whole session cost. Only traffic could, and traffic only talks to you if you route a slice of it first.
This part is the production runbook I promised at the end of Part 9: canary traffic splits, automatic fallback when a model degrades, and cost caps that stop a prompt regression from becoming a bill regression. The agent is the same e-commerce assistant from Parts 1 through 9: nine tools, conversation memory, the supervisor, and the human-in-the-loop checkout gate. I have been building production AI agents with Spring Boot and Spring AI for over a year, and every number below is from the rollout as I actually run it.
### Why rollout discipline is now the safety layer
Every gate in this series has lived before traffic. Part 6 proves the code is bug-free, Part 8 proves the answers are good on a fixed dataset, Part 9 proves a change beats its predecessor in a controlled comparison. None of them prove a change survives real users, because real users do things your dataset never imagined: they write messages in mixed Bengali and English, they ask about orders from three months ago, they argue with the agent about the cached price from Part 4.
The wider industry is moving in the same direction, and it makes the gap bigger, not smaller. Anthropic measured that Claude Code users approve 97% of permission prompts, a rate it says suggests most click through without reviewing each command, and in a 1,053-tester study its auto mode caught 89% of planted dangerous commands where humans caught 13.6%. It is making auto mode the default for new sessions on Pro, Max, and Team plans from August 14, per its own announcement. Whatever you think of that trade, it describes the same shift: agents act with less per-call human oversight, so the mechanics around the release decide what reaches users, not the review in the IDE. If your agent runs unattended, your canary and your fallback are the reviewer.
### Canary splits: Route a slice, watch it, trust it
The principle is boring on purpose. Two versions of the agent exist at the same time, both built from the same components as the production agent. A router sends a small percentage of conversations to the candidate and the rest to the baseline. You watch the candidate cohort against the baseline cohort, and you promote only when the candidate stops losing.
Spring AI gives you the two clients. The ChatClient reference shows the pattern I use: the auto-configured prototype ChatClient.Builder produces one bean per configuration, and you inject them by name with @Qualifier.
```java @Configuration public class AgentRoutingConfig { @Bean("baselineAgent") ChatClient baselineAgent(ChatClient.Builder builder) { return builder .defaultSystem(SYSTEM_PROMPT_V1) .build(); }
@Bean("candidateAgent") ChatClient candidateAgent(ChatClient.Builder builder) { return builder .defaultSystem(SYSTEM_PROMPT_V2) .build(); } } ```
Both beans share the same tool registry, the same memory wiring, and the same advisors as the production agent from Parts 1 through 9. The only difference is the system prompt, and in this agent the tool descriptions live inside the system prompt, so a tool-description change like the shipping prompt from Part 9 is a system-prompt change. If you change two things between the clients, the canary cannot tell you which one moved the numbers.
The router is where the discipline lives. The important detail is stickiness: the same conversation must stay on the same version for its whole life, because the agent's memory (Part 2) is per-conversation and per-version. A customer who asks a question, gets an answer from the candidate, then refreshes and hits the baseline, will experience a different agent mid-conversation. So I route on a hash of the conversation id, not on a per-message coin flip.
```java @Service public class CanaryRouter { private final Map<String, ChatClient> agents; private final CanaryProperties props;
public CanaryRouter(Map<String, ChatClient> agents, CanaryProperties props) { this.agents = agents; this.props = props; }
public ChatClient forConversation(String conversationId) { int bucket = Math.floorMod(conversationId.hashCode(), 100); if (bucket < props.candidatePercent()) { return agents.get("candidateAgent"); } return agents.get("baselineAgent"); } } ```
`agent.canary.candidate-percent=5` in application.properties, and a restart flips the split without a deploy. CanaryProperties is a small `@ConfigurationProperties(prefix = "agent.canary")` holder with a single int field, candidatePercent(), so the split comes from configuration, not code. That is the other rule: the ladder is 5, 10, 25, 50, 100, each step held for at least a day, and every step is a config change, never a code change. Code changes restart the experiment. I skip rungs only when the cohort numbers stay flat.
The candidate cohort is a cohort, not a sample. Compare the candidate against the baseline on the same slice of time: error rate, p95 latency, tool calls per conversation, refusal rate, and the Part 8 metrics sampled from live logs. The cohort comparison is what saved me on the shipping-prompt day. The nightly harness would have flagged the tool discipline drop the next morning. The canary flagged it at 6:42pm, hours after the 5% step, because the candidate's p95 had drifted from the baseline's by a margin the cohort report was built to catch.
Rollback is automatic and it is a config flip. My triggers: error rate exceeds the baseline by one percentage point for ten minutes, p95 exceeds 1.5x baseline for ten minutes, or any money-path conversation (checkout, refund, shipping) fails the Part 8 review. Any trigger sets candidate-percent to 0 and pages me. I do not want to be woken up to make a judgment call at 2am; I want to be woken up after the decision is made, to investigate.
The shipping prompt went back to the drawing board. The narrowed description, one that said the tool resolves a region and the delivery estimate and that it is called once per turn, re-ran the Part 9 gate, then climbed the ladder again. It took five days to reach 100%: two days at 5%, one at 10%, one at 25%, then straight to full, skipping 50 because the cohort numbers stayed flat. Traffic is the final reviewer, but it reviews one slice at a time.
### Fallback: Surviving a model that misbehaves
Canaries protect you from your own changes. Fallback protects you from everything else: a provider outage, a model that gets worse after an upstream update, a rate limit at peak hour. I split this into two failure classes, because they need different machinery.
Hard failures are exceptions: 5xx responses, timeouts, rate limits. The fix is a decorator around the model. Spring AI's ChatModel interface is small: call(Prompt) returns a ChatResponse, and stream(Prompt) returns a Flux<ChatResponse>. That interface is the seam. I wrap the primary model with a backup model and a small circuit state: three consecutive failures open the circuit for 60 seconds, during which every request goes to the backup, and a successful probe closes it again.
```java import org.springframework.ai.chat.model.ChatModel; import org.springframework.ai.chat.model.ChatResponse; import org.springframework.ai.chat.prompt.Prompt; import reactor.core.publisher.Flux; import java.time.Duration;
public class FallbackChatModel implements ChatModel { private final ChatModel primary; private final ChatModel backup; private final CircuitState circuit = new CircuitState(3, Duration.ofSeconds(60));
@Override public ChatResponse call(Prompt prompt) { if (circuit.isOpen()) { return backup.call(prompt); } try { ChatResponse response = primary.call(prompt); circuit.recordSuccess(); return response; } catch (RuntimeException ex) { circuit.recordFailure(); return backup.call(prompt); } }
@Override public Flux<ChatResponse> stream(Prompt prompt) { if (circuit.isOpen()) { return backup.stream(prompt); } return primary.stream(prompt) .onErrorResume(ex -> { circuit.recordFailure();
LOOP
LOOP
LOOP
LOOP
LOOP
LOOP
LOOP
LOOP
LOOP
LOOP