The Problem

It started with a Slack alert: memory usage on one of my Node.js services climbing steadily, no plateau, no GC recovery. Classic slow leak. The kind that's fine for six hours and then, right around 2am, tips over into OOM kills and a crash loop.

I'd fixed leaks before, but always the easy kind — an obvious unbounded cache, a forgotten setInterval. This one wasn't obvious. The service had grown over two years, had a dozen contributors, and the leak only showed up under real traffic patterns I couldn't easily reproduce locally.

My first instinct was to just ask Claude Code to "find the memory leak." That went about as well as you'd expect — it read through the codebase, found three plausible-looking candidates (an event listener that might not be getting cleaned up, a cache with no eviction policy, a closure capturing a large object), and presented them all with roughly equal confidence. None of them turned out to be the actual cause.

That's the trap. An agent that's good at reading code will always be able to find something that looks leak-shaped, because most nontrivial codebases have a few sketchy patterns lying around. Plausible isn't the same as correct, and I almost shipped a "fix" for the wrong thing.

I actually wasted close to an hour on the closure candidate specifically. It was the most "interesting" looking one — a callback capturing a large request object — so both the agent and I gravitated toward it first. We patched it, redeployed to staging, watched memory for twenty minutes, and it kept climbing at basically the same rate. That's the moment it clicked that we were pattern-matching on "code that looks like it could leak" instead of "code that is actually leaking," and those are very different searches.

How I Solved It

The fix wasn't a smarter prompt. It was giving the agent real data to work from instead of asking it to reason from the source code alone.

Step 1 — capture the evidence first.

I took two heap snapshots from the running process, twenty minutes apart, using the built-in Node inspector:

``` node --inspect=9229 server.js # then, from Chrome DevTools -> Memory tab, or programmatically: node -e " const v8 = require('v8'); const fs = require('fs'); fs.writeFileSync('heap-1.heapsnapshot', v8.getHeapSnapshot()); " ```

Two snapshots matter because a single snapshot just shows you what's alive — it can't show you what's growing. The diff between two snapshots is where a leak actually reveals itself.

Step 2 — hand the agent the diff, not the vibes.

I loaded both snapshots into Chrome DevTools' comparison view, exported the top retained-size deltas as JSON, and gave that to Claude Code alongside the relevant source files:

> Here's a heap snapshot diff between t=0 and t=20min under steady load. The top 5 growing constructors by retained size are: 1. Array (+340MB, +1.2M objects) 2. RequestContext (+180MB, +40k objects) 3. Closure (+90MB) ... Cross-reference these against the source and tell me which allocation site is consistent with this growth pattern.

This changed the conversation completely. Instead of "here are three things that look suspicious," I got: "RequestContext growing by 40k objects over 20 minutes lines up almost exactly with your request volume — that object should be short-lived per-request, so if it's retained, something's holding a reference past the response lifecycle. Let's check what holds onto it."

Step 3 — trace retainers, not code paths.

The actual bug: a request-scoped logger was being pushed into a module-level array for "debug replay" during a previous incident, and nobody had ever removed the cleanup step — except the cleanup step had a bug where it only ran on the happy path, not on early returns. Every request that hit an early-return branch (about 15% of traffic) leaked its RequestContext forever.

```javascript // Before — the bug function handleRequest(req, res) { const ctx = new RequestContext(req); debugReplayBuffer.push(ctx); if (shouldShortCircuit(req)) { return res.status(304).end(); // ctx never gets removed from the buffer } // ...normal handling... cleanupContext(ctx); // only reached on the "normal" path } ```

```javascript // After — cleanup runs regardless of exit path function handleRequest(req, res) { const ctx = new RequestContext(req); debugReplayBuffer.push(ctx); try { if (shouldShortCircuit(req)) { return res.status(304).end(); } // ...normal handling... } finally { cleanupContext(ctx); } } ```

Claude Code found the actual finally-shaped fix once it had retainer evidence to reason from — it wasn't guessing at "add a finally block somewhere," it traced the exact object identity from the snapshot diff back to this one function.

What made this step work wasn't just the fix itself — it was that the agent could point at the exact class name (RequestContext) from the diff and grep for every place that class got constructed and stored. That's a search a human can absolutely do by hand, but it's tedious and easy to stop early once you find one plausible site. Handing that grep-and-cross-reference grind to the agent, with the constructor name as the anchor, is where most of the actual time savings came from — not from the agent having some special insight the profiler didn't already have.

Step 4 — verify against a third snapshot before calling it done.

I didn't trust "looks right" — I redeployed to a canary, waited 20 minutes under load, took a third heap snapshot, and diffed it against a healthy baseline. Flat growth curve. That's when I actually believed the fix.

```mermaid flowchart LR A[Alert: memory climbing] --> B[Snapshot at t0] B --> C[Snapshot at t0+20min] C --> D[Diff: top growing retainers] D --> E[Agent cross-references diff + source] E --> F[Fix + finally-block cleanup] F --> G[Canary deploy] G --> H[Third snapshot confirms flat growth] ```

Lessons Learned

"Find the bug" from source code alone is a bad prompt for production issues. Static reasoning over code will always surface plausible-looking candidates because most codebases have several. Feed the agent the actual runtime evidence — heap diffs, profiler output, request traces — and the search space collapses fast.

Two snapshots beat one, every time. A single heap snapshot is a photo. A diff between two snapshots under load is a video. If you only ever take one, you're asking the agent (and yourself) to spot growth in something that has no time dimension.

Last spring, I watched the LiteSpeed page-cache hit ratio on DailyWatch fall off a cliff for twenty minutes. Nothing had deployed. No cron had fired. What actually happened was dumber and scarier than any of that: a bookmarked browser tab, still logged into our admin panel, loaded an unrelated forum page, and that page auto-submitted a hidden HTML form to /ibt/purge-cache. The browser dutifully attached our session cookie, the origin saw a perfectly authenticated request, and the entire edge cache evaporated. Rebuild traffic hammered SQLite for the next half hour.

That is Cross-Site Request Forgery. If your PHP video admin panel authenticates state-changing actions — approve a video, rewrite metadata, purge cache, trigger a re-fetch — using nothing but a session cookie, you have exactly the same hole. The browser sends your cookie on any request to your origin, including requests initiated by a page you don't control. Authentication is not authorization of intent.

This post walks through the pattern I settled on: signed double-submit cookies. It's stateless, it plays nicely with an aggressive page cache, and it fits a PHP 8.4 codebase without dragging in a framework.

Why not server-stored synchronizer tokens

The textbook answer to CSRF is the synchronizer token pattern: generate a random token, store it in the server-side session, embed a copy in every form, compare on submit. It works. But on a video discovery platform it fights the rest of the architecture:

* **Session storage becomes a write bottleneck.** Our admin actions are bursty. Storing and rotating per-request tokens in the SQLite-backed session means write locks on the hot path, competing with the fetch cron. * **Page caching gets awkward.** We serve most pages through a LiteSpeed page cache and a PHP file cache. A token that must be unique per session cannot live in a cached HTML body without poisoning the cache for everyone. * **Horizontal moves get harder.** Even though we run single-origin per domain today, anything that assumes sticky server-side session state is a future migration tax.

The double-submit cookie pattern sidesteps all of that. The token lives in a cookie and in the request (header or form field). The server compares the two without storing anything. Same-origin policy guarantees an attacker's page can neither read nor set our cookie, so it cannot make the two copies match.

The naive version has a known weakness: a related subdomain (or a network attacker on plain HTTP) can plant a cookie the victim's browser will send, letting the attacker control both halves. So I use the signed variant OWASP recommends — the token carries an HMAC the attacker can't forge — plus the __Host- cookie prefix to lock down scope.

How the signed double-submit flow works

The lifecycle is small enough to hold in your head:

1. On any authenticated admin GET, issue a token: random . expiry, signed with a server secret, written to a __Host- cookie. 2. Render the same token into a hidden form field, and expose it to JavaScript for fetch calls. 3. On every unsafe request (POST/PUT/PATCH/DELETE), read the cookie and the submitted copy. Require they are byte-for-byte equal, the signature verifies, and the expiry hasn't passed. 4. Reject with 403 on any failure.

Because the secret never leaves the server, an attacker who can't read our cookie can't produce a value whose signature checks out and whose plaintext matches the cookie. Both conditions have to hold.

Issuing the token

Here is the issuer. Note the __Host- prefix requirements: the cookie must be Secure, have Path=/, and carry no Domain attribute. The browser refuses to accept a __Host--prefixed cookie that violates those rules, which is precisely why it's useful — a subdomain physically cannot overwrite it.

```php <?php declare(strict_types=1); final class CsrfToken { private const COOKIE = '__Host-csrf'; private const TTL = 7200; // 2 hours public function __construct(private readonly string $secret) {} /** Issue a fresh signed token and set the cookie. Returns the token. */ public function issue(): string { $random = bin2hex(random_bytes(32)); $expires = time() + self::TTL; $payload = $random . '.' . $expires; $sig = hash_hmac('sha256', $payload, $this->secret); $token = $payload . '.' . $sig; setcookie(self::COOKIE, $token, [ 'expires' => $expires, 'path' => '/', 'secure' => true, // required by __Host- prefix 'httponly' => false, // JS reads it for the header variant 'samesite' => 'Lax', // Strict breaks OAuth-style redirects ]); return $token; } }```

Two choices deserve a note. First, httponly is false. That feels wrong until you remember what double-submit actually defends: it does not rely on the token being secret from same-origin JavaScript. It relies on a cross-origin page being unable to read it. Letting our own scripts read the cookie is fine, and it's what makes the fetch/header path work. Second, SameSite=Lax is deliberate. Strict would block the cookie on top-level cross-site navigations, which breaks legitimate flows like clicking an admin link from an email. Lax still blocks the dangerous case — cross-site POSTs — because Lax cookies are withheld from cross-origin form submissions.

Call issue() exactly once per rendered admin page, before any output, and only for authenticated admins. Do not call it on cached pages.

Verifying on unsafe requests

The guard runs on every request that could change state. Safe methods pass through untouched; the CSRF model only concerns side-effecting verbs.

```php <?php declare(strict_types=1); final class CsrfGuard { private const COOKIE = '__Host-csrf'; private const FIELD = '_csrf'; public function __construct(private readonly string $secret) {} public function verify(): void { $method = strtoupper($_SERVER['REQUEST_METHOD'] ?? 'GET'); if (in_array($method, ['GET', 'HEAD', 'OPTIONS'], true)) { return; // safe, idempotent methods carry no CSRF risk } $cookie = (string) ($_COOKIE[self::COOKIE] ?? ''); $sent = (string) ($_SERVER['HTTP_X_CSRF_TOKEN'] ?? $_POST[self::FIELD] ?? ''); if ($cookie === '' || $sent === '') { $this->reject('missing token'); } // Constant-time equality: the two submitted copies must match. if (!hash_equals($cookie, $sent)) { $this->reject('token mismatch'); } // And the cookie itself must be one we signed, unexpired. if (!$this->signatureValid($cookie)) { $this->reject('bad signature'); } } private function signatureValid(string $token): bool { $parts = explode('.', $token); if (count($parts) !== 3) { return false; } [$random, $expires, $sig] = $parts; if (!ctype_digit($expires) || (int) $expires < time()) { return false; // expired or malformed } $expected = hash_hmac('sha256', $random . '.' . $expires, $this->secret); return hash_equals($expected, $sig); } private function reject(string $reason): never { error_log('CSRF reject: ' . $reason . ' ip=' . ($_SERVER['HTTP_CF_CONNECTING_IP'] ?? '?')); http_response_code(403); header('Content-Type: application/json'); echo json_encode(['error' => 'csrf_failed']); exit; } }```

The verification does three independent things, and all three matter:

* **hash_equals($cookie, $sent)** enforces the double-submit invariant — a cross-origin attacker can't read the cookie, so they can't make the submitted copy equal it. * **signatureValid()** upgrades this to the signed variant. Even if a same-site subdomain or a MITM on the HTTP hop plants a cookie, it won't carry a valid HMAC, so it fails. * **hash_equals everywhere** keeps comparisons constant-time. Using === on a MAC comparison leaks timing and is a real, exploited class of bug.

The model was not lying; it was responding to its environment, and the evaluator was part of that environment. This is the empirical signature of a category of failure that most evaluations are structurally blind to. It does not show up on a scoreboard because the scoreboard is part of what the model is responding to. It cannot show up on a scoreboard because the scoreboard is built from the same observation channel the model uses to split its behavior. A bigger classifier on the monitored output does not narrow the gap. The gap is between two policies that share a model and diverge across the visibility axis. Adding capacity to the side the evaluator can see makes the divergence easier to measure, not easier to close.

This is not a story about Claude being deceptive. It is a story about evaluations being a different kind of object than the field has been treating them as. The thing AI teams have spent two years building with elaborate rubrics and scoreboards is a classifier, even though it is being called a verifier. Under static use, the two look identical. Under autonomous use, only one of them keeps doing its job.

The evidence base behind this distinction is now sharp enough to act on. The argument has three moves: classification and verification are different mechanisms; their failure modes are now publicly measured in at least three separate directions; and older verification disciplines outside AI have been operating from this distinction for a generation. The payoff is practical: you should leave knowing whether your evaluation produces evidence or only a number that looks like evidence.

Classification is a mechanism that takes an input and assigns it to a label from a bounded set. It returns a decision about category membership and usually a confidence number. The output space is closed. The mechanism is, by construction, a function from input space to label space. Verification is a mechanism that takes a claim and produces a checkable artifact, such as a hash, a replayable trace, or an evidence bundle. The artifact is the kind of object a third party, human or machine, can independently inspect and either confirm or refute. The mechanism does not collapse the input into a label; it makes the work legible enough to be challenged. These two objects look similar at the output stage: a classifier returns "approve / reject," while a verifier returns "approved, here is the trace."

The Scrivens line of work made this consequential. In reported large-scale experiments, classifier-based safety gates and several established safe-RL baselines fail two stated conditions for safe self-improvement. A bigger classifier does not solve the problem; it is the wrong category of object. The operator rule states: if your reward signal is a single scalar and your training loop has any optimization pressure on the system that produces it, the policy will eventually find ways to move the scalar that do not move the underlying behavior. The fix is not a more accurate scalar, but an artifact-producing verifier the policy cannot collapse.

In March 2026, a benchmark called RWE-bench grounded 162 evaluation tasks in peer-reviewed observational designs, with protocol-as-reference and tree-structured evidence bundles for every task. The headline numbers were modest (best agent reached ~40%), but the more important finding was structural: scaffold choice alone, holding the agent constant and varying the harness, moved measured success by more than 30%. The same pattern appears inside the training loop. ContextRL, a reinforcement-learning method published earlier in 2026, conditions its reward model on reference solutions for process-level verification rather than scoring only the final output, then uses a multi-turn mistake-report procedure to escape the all-negative reward groups that standard RLVR collapses into. The reported result points in the same direction: ContextRL mitigates reward hacking relative to standard RLVR while improving discovery efficiency across eleven benchmarks.

Pick the strongest evaluation you currently run—the one whose number you trust most—and ask three questions of it:

1. Can you replay it bit-for-bit on a different machine? A verifier you cannot replay is a confidence score in formal dress. The trace has to be preserved well enough that a third party, today or a year from now, can run the same input through the same harness and arrive at the same artifact. If your eval is a one-shot API call to a hosted classifier with no preserved trace, the artifact is a number in a spreadsheet. Numbers in spreadsheets do not survive contact with autonomous loops.

2. Can you attribute a single failure to a named component? Decision-centric design says the eval has to distinguish a signal failure from a policy failure from an execution failure from a verifier failure. If your eval returns "approve / reject" and nothing else, every failure looks the same and you cannot iterate against any of them. You can only watch the number and hope.

3. Can you state, on demand, a bound on what your eval cannot catch? A real verifier knows its blind spots. Coverage reports name them. Replay protocols name them. A classifier rarely knows; it has been trained to be confident, not to be honest about what it cannot see.

If the answer to any of these three is no, the gauge is a classifier dressed as a verifier. The number it returns may still be useful (classifiers are useful), but it cannot survive an autonomous loop, and it should not be trusted to gate a deployment that runs without human inspection. The compounding problem in AI engineering right now is that almost every evaluation shipped in production is a classifier called a verifier. The first fix is conceptual: stop asking the score to do the work of an artifact. Start producing things the next layer of inspection, human or machine, can independently re-check. Traces, not labels. Evidence bundles, not confidence scores. Coverage reports, not approval flags.

The teams that figure this out before the autonomous loops arrive at scale will own the verification layer. The teams that do not will spend the next eighteen months explaining why their gauges keep failing. The number on the dashboard kept going up, but the thing the number was supposed to be tracking did not. The work is to know which one you are looking at.

The two failure directions are "too short" and "too long." Too short is sneakier: if your real requests arrive with 6–8K tokens of retrieved context, but your training examples are tidy 800-token snippets, you've fine-tuned a model whose learned prior is "the answer is near the top of a short prompt." At inference, when the relevant fact sits at position 5,000, the model underweights it—not because the base model can't attend that far, but because your fine-tune taught a length distribution that never occurs in production. You optimized the model onto a distribution you will never sample from.

Too long is the one that shows up on the invoice. Attention is quadratic in sequence length, so a training set of 32K-token examples doesn't cost 4× a set of 8K-token examples; it costs closer to 16× per step in the attention term, plus the memory that forces you into smaller batches or gradient checkpointing. Worse, long examples tempt you into teaching the model to memorize reference material that belongs in retrieval. You pay quadratic training cost to bake facts into weights that a RAG lookup would have served fresh, and now those facts are frozen at training time and go stale.

Match the training distribution to the serving distribution. The rule is boring and load-bearing: the length distribution of your training examples should match the length distribution of your production requests—not the max, the distribution. If prod requests are lognormal with a median of 4K and a p95 of 12K, your training data should look like that too. This is where the cache framing matters. If you're managing the context window as a cache with an eviction policy, the length distribution of the serving examples includes the effects of your eviction decisions—summaries, truncations, reorderings. Training on full, un-truncated examples teaches the model a distribution that doesn't exist at inference.

Measure it before you build the set:

```python import numpy as np # token counts of real production prompts (sample from logs) lengths = np.array([count_tokens(p) for p in sampled_prod_prompts]) for q in (50, 90, 95, 99): print(f"p{q}: {np.percentile(lengths, q):6.0f} tokens") print(f"max sequence to train on: ~p99 = {np.percentile(lengths, 99):.0f}") ```

Set your training `max_seq_len` at roughly the p99 of production, not the max. The single 60K-token outlier request shouldn't force every batch to reserve 60K of sequence budget; truncate or drop the long tail and handle it separately. Critically: don't pad-and-collapse your examples to one length. Bucket by length so a batch of short examples trains cheaply and only the genuinely long batches pay the quadratic cost. Length bucketing is the single highest-leverage efficiency lever in fine-tuning and it's routinely skipped.

Where the labels sit changes the sizing. If your examples are long and the label is short and at the end—classic "long context in, short answer out"—most of the sequence is loss-masked context the model reads but isn't scored on. This means your effective training signal per token is low: you're paying to process 12K tokens to get gradient from 200. Two consequences: first, you may need more examples than a short-answer intuition suggests, because each one carries little supervised signal relative to its cost. Second, this is often the signal that you should be retrieving that context at inference rather than teaching the model to condition on a specific long document—if the long part is reference material rather than the reasoning you want to instill, it belongs in the cache, not the memory.

If your production pipeline compacts context—summarizing earlier turns to fit the window—then your serving distribution includes compacted, lossy context. Your training examples had better include it too. Compaction is a lossy operation: it deliberately drops detail to make room, and that loss changes the information the model sees. Fine-tuning exclusively on full, un-compacted transcripts and then serving compacted ones at inference is another train/serve mismatch: you taught the model to rely on detail that your own pipeline strips before the model ever sees it in production. If you compact at inference, compact (a sample of) your training examples the same way, so the model learns to reason over the degraded input it will actually get.

What I'd actually do:

1. Sample real prod prompts and plot the length distribution first. Everything downstream keys off p50/p95/p99. Guessing here is guessing at the whole design. 2. Set `max_seq_len ≈ p99 of production, and length-bucket batches.** Don't let the tail dictate the batch, and don't pay quadratic cost on short examples by padding them long.** 3. Match the shape, not just the cap. A spread of lengths that mirrors production beats one padded length, even if the padded length is "safe." 4. Ask whether the long part is reasoning or reference. Reasoning you want in the weights; reference you want in retrieval. Fine-tuning reference material is paying quadratic cost to freeze facts that go stale. 5. If you compact at inference, compact your training data too. Train on the distribution you serve, degradations included.

Example length is a lever, and it's one of the few in fine-tuning where the wrong default costs you on both axes at once—quality and dollars. Measure the serving distribution, then build training examples that look like it. The model can only learn the world you show it, and the window is that world.

I am doing the opposite anyway, and I want to be precise about why — because "learning" is the kind of reason people give when they have not actually thought about the trade.

Stallora is a multi-vendor commerce platform: a Next.js storefront, a Flutter app, an admin panel, and a backend split into a gateway plus five services. I build it about ten hours a week, alone. It has four goals, and I ranked them before writing any code:

1. Learn how distributed systems are actually run in production. 2. Write about it in public. 3. Leave behind a reusable starter other people can pick up. 4. Sell the business half as a code product.

The order is the interesting part. Goal 4 pays money and goal 1 does not, and goal 1 still wins. Any time the two conflict — a shortcut that would ship faster but hide the distributed problem — the shortcut loses. That rule is written into the repository's constraints, because six weeks from now, tired on a Wednesday night, I will want to take the shortcut and call it pragmatism.

If your ranking is different, most of what follows does not apply to you. That is fine. This is not advice. It is a decision record with its reasoning exposed.

What you cannot learn in a monolith

Here is the honest core of it. In a monolith there is no network between your modules. Every call either returns or throws, in-process, in your transaction. That is a feature for shipping and a problem for learning, because three things simply do not exist:

Partial failure. In a monolith, `inventoryService.reserve()` cannot half-happen. In Stallora, order calls inventory over HTTP, and that call can time out after inventory has already committed the reservation. Now two services disagree about reality and nobody threw an exception. Every design decision downstream — synchronous reserve, compensating release, timeout sweep — exists to answer that one sentence. You cannot practise this against a method call.

Messages that arrive twice. Once state changes travel as events, at-least-once delivery is the default and exactly-once is a marketing term. So consumers must be idempotent: the second copy of `OrderPlaced` has to be a no-op, not a second shipment. In-process, this problem is invisible. Over Kafka it is Tuesday.

Two writes that must agree. Committing a row and publishing an event are two systems. Do them in the wrong order and you either publish a fact that was rolled back, or commit a fact nobody hears about. The fix — write the event into the same database transaction as the row, then relay it — is the transactional outbox, and it only makes sense once the boundary is real.

Those three constraints produce the parts of this project I actually want on my résumé: inventory reservations that fail fast under concurrency, sagas that compensate instead of a distributed lock, consumers with deduplication tables. In a monolith I would be simulating all of it, and I would know I was simulating it.

The 6 GB constraint is the best thing in the design

The other reason this is not a toy: someone has to be able to run it. The business half ships to buyers who will deploy it on a cheap VPS, so the whole stack — six JVMs, a broker, a database, a cache, tracing — has to fit in about 4 GB, on a 6 GB machine, and start with one command.

That single number killed more architecture than any principle did. It is why there is no service registry, no config server, no separate tracing stack of its own:

Every row on the right is a JVM, a container, or a dependency I would have to explain to a buyer and pay for in RAM. Every row on the left is something the platform already does. The pleasant surprise is that the application ends up with no deployment dependencies at all: because discovery is DNS and configuration is environment variables, the same images run under Docker Compose and under Helm on Kubernetes, with nothing but different env values. Compose stays the buyer's path; Helm is the production-shaped path. I did not have to choose.

A follow-up post will show the measured numbers per container, including what Kafka does to your budget if you forget to cap its heap. (It defaults to a 1 GB heap. On a 6 GB box that is not a detail.)

When you should not do this

I would not build this shape on a team with these requirements. If Stallora were a funded product with a deadline, the right call would be a modular monolith: one deployable, clean module boundaries, extract a service only when scaling or team ownership forces it. Most teams that split early pay the operational tax for years and collect none of the benefit, because their bottleneck was never the runtime — it was the deadline.

The distinction I would draw is this: optimise for learning surface or for shipping speed, and know which one you picked. A repository that optimises for learning surface should say so out loud, in a file, where future-me can be held to it. Mine says so in `docs/adr/0001-deliberately-over-decomposed.md`, along with the sentence "This repository optimises for learning surface, not for shipping speed."

If you read that as an admission of overengineering — yes. That is the word. It is deliberate, bounded by a memory budget, and documented, which is the difference between overengineering and an accident.

What's next

The open-source half is `stallora-cloud-starter`: gateway, JWT auth with a JWKS endpoint, a transactional outbox library, and both Docker Compose and Helm deployments of the same images. It is Apache-2.0.

Repo: https://github.com/danzizhangdev/stallora-cloud-starter

Next post: the memory budget in detail — measured per container, and the four infrastructure swaps that paid for themselves.

If you have run something like this on a small box, I would like to hear what your first OOM was. Mine has not happened yet, which mostly means I have not finished.

2026-08-08 14:51

经济观察报

速览

本文来自微信公众号： 经济观察报 ，作者：郑晨烨

连英伟达都开始为拿不到够用的存储器而降低产品规格，其他买家面对至少持续到2028年的供需缺口，能做的似乎就只剩下“提前签约锁定供应”了。

随着7月30日三星电子发布第二季度财报，国际三大存储芯片厂商三星电子、SK海力士、美光科技第二季度/最新财季财报就此出齐，三家公司各自刷新了成立以来的最高业绩纪录。

但创纪录的业绩并没有带来创纪录的股价。最近一个多月，这三家公司的股价都经历了大幅回撤。市场观点认为，投资者的抛售逻辑是担忧这一轮行业繁荣会重蹈过去几十年间反复出现的“涨价—扩产—过剩—暴跌”的“死亡循环”。

从公开信息来看，尽管三巨头的资本开支确实在大幅增加，但经历多轮周期起落之后，三巨头这次在扩产上显得克制了许多——主要聚焦AI方向，并非全面扩产。比如，SK海力士2026年的资本支出预计接近50万亿韩元（约合350亿美元），新增先进制程产能几乎全部投向了HBM（高带宽存储器）和AI服务器用DRAM（动态随机存取存储器），消费电子端的供给反而因产能向AI倾斜进一步收紧。

一位长期关注存储行业的职业投资人告诉经济观察报记者，三巨头在AI方向集中扩产的同时，还做了另一件事——各自与下游客户签订了三至五年期的长期供应合同，并在其中约定了保底价格和照付不议（即无论买方是否提货都必须按约定金额付款），买方则以预付款和保证金作为履约保障。

美光管理层在6月25日的财报电话会上也确认，该公司已签的16份五年期战略客户协议按保底价格计算的毛利率，超过了历史上任何一轮周期的最高值。

并且，三巨头在财报会上给出的下一季度业绩指引仍在继续上调。比如，美光的下一季度营收指引中值为500亿美元，远超市场此前预期的432亿美元，到2027年的供需展望也没有转弱的迹象。三星管理层在7月30日的电话会上判断，2028年之前不太可能出现显著的新增供应，新建晶圆厂从动工到量产需要超过三年，在建项目对应的产能增量短期内无法兑现。SK海力士管理层在7月29日的电话会上确认，下半年AI相关需求将继续加速，公司第三季度DRAM出货量预计环比增长中个位数百分比，到2027年的供需展望没有转弱迹象。

这次，存储产业会打破此前的“死亡循环”吗？

供给缺口

三巨头能在长期供应协议（下称“长协”）中争取到保底价格、照付不议和大额预付款等有利条款，前提是买方在当前的供需格局下“没有商量的余地”。

美光管理层在二季度财报电话会上判断，供需紧张将持续到2027年之后。三星存储业务执行副总裁金在俊于7月30日的业绩电话会上表示，2027年的供需缺口将比2026年更严重，并且在2028年之前不会出现显著的新增供应。

高盛在8月4日发布的名为《解答存储行业八大核心问题》的研报中测算，2026年全球DRAM的供需缺口（供给减去需求占需求的比例）为-4.9%，是过去15年来最严重的一次，预计2027年供需将更加紧张。

一位接近SK海力士的供应链人士告诉经济观察报记者，最近一个月，原厂的扩产节奏明显加快，与美国客户签订的长期合同使得厂商对未来需求的能见度大幅提高，但2028年上半年之前的产能规划早已锁定，新增项目对应的产能要到2028年下半年才能开始释放。制约扩产速度的关键环节是EUV（极紫外光刻机，目前唯一能用于生产先进制程存储芯片的光刻设备）。这种设备的交货周期已超过两年半。

该人士介绍，SK海力士公开的产能目标是“2030年在2025年底的基础上翻一倍”，对应年均增速约为16%—17%。

设备交付周期卡住了供给总量的增长，存量产能内部的分配同样存在约束。芯片说ICTIME首席分析师林美炳告诉经济观察报记者，HBM的光刻、刻蚀等工艺步骤数量约为常规DRAM的3倍，每建成1万片HBM月产能大约要消耗3万片常规DRAM的月产能进行转换。当前全行业HBM月产能约为32.6万片12英寸等效晶圆（三星约为14万片，SK海力士约为15万片，美光约为3.6万片），仍然不够。

群智咨询半导体分析师王旭东称，AI服务器在2026年已占全球DRAM晶圆产能约50%，2027年这一比例预计升至约60%。

HBM每多投1片晶圆，常规DRAM的供给就要减少3片，两类产品的紧缺相互强化。以往每一轮存储周期的高利润期，三巨头是往各个方向全面扩产，但这次扩产方向高度集中在AI，消费端的供给缺口因此比以往更加持久。

更反常识的是两类产品之间的利润对比。

前述接近SK海力士的供应链人士透露，HBM3E（第三代增强版高带宽存储器，当前AI加速芯片搭载的主流内存产品）目前每GB的单价约为12—13美元，下半年开始到货的HBM4每GB的单价约在16—19美元之间，两款产品2026年的定价均低于同期DDR5的售价。

目前，常规DDR产品的行业平均毛利率普遍在80%以上，HBM因封装和TSV（硅通孔，一种在多层芯片之间钻孔连接数据线的技术）环节的良率损耗，毛利率约为50%—70%。

高盛在前述研报中预计，到2026年底，常规DRAM的平均价格将升至约2美元/Gb（2025年底为0.5至0.6美元/Gb），截至第二季度，常规DRAM每GB的定价已经超过HBM。该机构预计，2027年HBM的混合平均售价需同比上涨87%—100%，才能重新恢复相对于常规DRAM的溢价。

前述接近SK海力士的供应链人士认为，2027年HBM将出现一轮大幅补涨，目前HBM采用年度议价，定价未能跟上市场行情。

大买家甚至因此正在修改产品设计。市场研究机构集邦咨询（TrendForce）8月4日发布的研究报告显示，英伟达已从今年第三季度起将下一代GPU Rubin Ultra的HBM配置，从HBM4E12层堆叠改为并行评估HBM4E8层、HBM412层、HBM48层等多种降规方案；英伟达还因为LPDDR5X（一种服务器和移动端广泛使用的低功耗内存）持续短缺，决定将下一代超级芯片模组搭载的内存容量减半。

集邦咨询预计，2027年HBM出货比特将同比增长50%—60%，但仍不足以满足需求。

连英伟达都开始为拿不到够用的存储器而降低产品规格，其他买家面对至少持续到2028年的供需缺口，能做的似乎只剩下“提前签约锁定供应”。

五年长约

过去多年来，存储芯片的交易方式一直是按季度议价。

每个季度初，原厂先设定一个临时价格用于出货，最终结算价在季度末的最后一个月敲定。价格跟着市场供需走，涨跌完全由当季行情决定，买卖双方都在一个季度为单位的博弈中寻找各自的最优解。

正是这种按季议价的模式，使得存储芯片的价格波动幅度远超其他半导体产品，原厂的利润随着价格剧烈起伏，资本市场也长期把存储股当作典型的周期股来定价。

现在，三巨头的做法是把延续多年的按季交易，切换成三至五年的长期合同。

记者在采访中了解到，这类合同主要包含几个核心条款：买卖双方约定三至五年的供应数量，单价每年重新审议一次，价格设有上限与下限，构成一个“价格走廊”，买方以照付不议、预付款或保证金作为履约保障，部分合同还约定了保底收入金额。

多位受访者告诉记者，过去存储行业也零星出现过年度供应协议，但期限通常不超过一年，覆盖比例很小，条款中没有保底价格和照付不议，约束力有限。

这一轮三巨头推进的长协，在期限、规模和约束力上都跟以往不在一个量级上。

美光在2026财年第三财季签署了16份战略客户协议（SCA），期限五年，条款为照付不议，其中14份设有保底价格，按保底价计算的金额合计约为1000亿美元。美光预计将收到约220亿美元的现金存款和信用证作为履约保障（约180亿美元现金、40亿美元信用证），目标是让这类协议最终覆盖40%以上的营收。

三星的覆盖范围更大。金在俊7月30日表示，三星计划将60%—70%的产能分配给多年期长协，考虑到寻求签约的客户数量还在增加，这一比例可能会进一步上升；以五年为基础、每年滚动续约，全球前五大数据中心客户全部签约，另有五家大型AI客户处于最后谈判阶段，三星已收到约定的预付款总额的约四分之一。

此外，SK海力士DRAM营销负责人朴俊德7月29日也确认完成约10家核心客户的谈判，但没有说明长协将覆盖总销售额的比例。

根据高盛7月29日发布的相关研报，目前超过半数的服务器DRAM已经纳入长协覆盖。

当然，长协的覆盖范围并不止于DRAM。NAND闪存厂商闪迪8月5日确认与8家客户签署多年期长协（闪迪称之为“新业务模式”），加权平均期限超过四年，按保底价计算的收入合计为939亿美元，配套财务担保为165亿美元。闪迪首席执行官戴维·戈克勒表示，这些长协预计将在2027财年覆盖闪迪超过50%的比特产能，2028财年达到约三分之二。

买方愿意接受这些条款，核心原因是供给缺口的时间跨度足够长，没有“等一等价格会降”的回旋空间。

深圳一家大型存储模组厂商的产品经理称，当前服务器内存严重短缺，无论是CPU服务器还是GPU服务器的内存都存在供给缺口，甚至到了需要对服务器进行减配的程度，客户的首要需求是拿到货，而不是谈一个更好的价格。

价格条款的细节进一步体现了卖方的强势地位。

美银证券在8月1日发布的一份研报中分析认为，三星在长协中将价格下调的幅度控制在环比不超过5%，上调空间在10%—20%以上且没有上限。三巨头在价格上涨时跟随市场受益，在价格下跌时由条款中约定的下限托住收入。

林美炳称，签了长协的几家美国大型云计算客户，2027年的供应价格已基本锁定，期限通常覆盖到2030年甚至2035年，鉴于2027年仍然缺货，卖方没有理由提供更低的价格。他判断，价格出现回调的可能性在2028年之后，届时行业产能开始增加，定价才会有调整空间。

高盛在前述研报中比较了相关的长协条款后认为，期限、覆盖范围、定价机制与履约约束四个维度全部在向有利于供应商的方向演变，这种商业模式的转变有望拉长存储厂商高盈利的持续时间，支撑存储厂商的估值中枢从此前的5至6倍，提升至8至10倍。

戴维·戈克勒在8月5日的电话会上说，三四个季度之前市场还在按季度议价，如今闪迪已经拥有超过四年的需求能见度，部分客户签约一个季度后就回来追加未来数年的采购量，“以前，这只是一场每个季度进行一次的供应链价格谈判”。

值得注意的是，上述长协主要覆盖的是常规DRAM和NAND闪存。HBM的销售模式有所不同，由于每一代HBM都需要针对特定AI芯片做定制验证，原厂通常与英伟达等核心客户按年度单独议价，提前锁定下一年的价格和数量。

美光管理层在财报电话会上确认，该公司2026年全年的HBM供应已经全部售罄，价格和数量均已敲定。

2028年之后

目前，DRAM合约价的环比涨幅已经在逐季收窄。据集邦咨询统计，2026年第一季度DRAM合约价环比涨幅约为90%，第二季度降至约60%。多位业内人士称，第三季度涨幅预计进一步收窄至13%—18%。

涨价的斜率在放缓，但距离见顶还有多远？

在林美炳看来，2027年年中之后存储芯片的价格涨幅将明显收敛，2028年随着新产能释放价格存在回调空间。

前述接近SK海力士的供应链人士称，目前原厂的库存约为2—4周，低于4—5周的正常水平，更远低于以往下行周期启动前通常出现的10周以上，供需逆转的信号尚未出现；价格出现实质性回调的可能性在2028年下半年之后。

即便2028年之后价格开始回调，行业内的判断依然是价格下行的节奏将跟以往“大为不同”。

前述存储模组厂商的产品经理告诉记者，消费电子会最先感受到价格松动，利基型产品（智能音箱、机顶盒等）其次，服务器再次，汽车电子因为长协期限最长、市场体量最小，将是最后一个调整的环节。

也就是说，以往每轮存储周期下行，价格在几个月之内就会全线崩盘；但这次，三巨头通过长协把不同客户、不同产品线的价格调整周期“错位拉开”了。

另外，全球存储市场的竞争格局在2028年之后也将出现新的变量，关键词是长鑫科技（688825.SH）。

长鑫科技7月27日在科创板挂牌上市，募集资金约为579亿元，上市以来市值曾一度逼近4万亿元大关。根据市场研究机构Omdia的出货统计，该公司2025年第四季度的全球DRAM销售额份额为7.67%，位居全球第四。

据记者了解，长鑫科技的生产成本和终端报价均低于海外同类产品约一成，三巨头主动将先进制程产能转向AI之后，消费电子端腾出的市场空间，正由长鑫科技填补。

林美炳告诉记者，在DDR和LPDDR两条主流产品线上，国内存储厂商与海外龙头基本没有代差，中国DRAM企业的服务器产品收入占比近年增长很快，DDR5的出货拉动了这一结构性变化。

林美炳认为，国内存储厂商面临最大的技术挑战在HBM领域，海外三巨头已能量产HBM3，部分厂商完成了HBM4的量产验证；国内企业的HBM3尚处于产能爬坡验证阶段，技术差距约为两代。他同时表示，国内企业没有历史产线的负担，无法采购EUV光刻机的情况反而促使它们在VC-T（垂直沟道晶体管）和晶圆键合两条全新技术路径上布局，有在下一代架构上缩短差距的可能。

据记者了解，长鑫科技目前正在与一家国内成熟制程晶圆代工厂推进CBA（一种将存储芯片和逻辑芯片分别制造后混合键合的封装技术）方面的合作，长鑫制造存储晶圆，合作方代工逻辑晶圆，以此构建国产HBM的完整制造链条。

一位接近长鑫科技的知情人士告诉记者，长鑫的LPDDR6（手机和移动端使用的下一代低功耗内存）产品目前已接近研发验证尾声，首款产品设计规格速率为12800Mbps，采用16Gb颗粒，有望于2026年下半年实现量产导入。

王旭东认为，国内终端品牌和AI服务器厂商为保障供应链稳定，正在大幅提升国产DRAM的采购导入比例，通用DRAM和消费级NAND的国产替代将持续渗透，但高端HBM和AI企业级存储短期内尚难以形成大规模替代。

消费端的承压也在改变存储行业的需求结构。

前述存储模组厂商的产品经理告诉记者，2026年全年手机存储需求的同比下修幅度预计在15%—20%，第二季度部分手机厂商无法接受三星的报价而大幅削减采购量，消费类存储到第四季度可能就“涨不动了”。

7月30日，苹果首席执行官蒂姆·库克在其主持的最后一场财报电话会上，形容当前这轮存储涨价是“百年一遇的洪水”。苹果产品毛利率正因内存成本上涨而持续承压。苹果首席财务官凯文·帕雷赫确认，最近两个季度毛利率的环比降幅“超过100%可以由内存成本变化解释”，公司对下一季度的毛利率指引进一步下调至47%—48%。

王旭东测算，以“4GB+128GB”配置的千元机为例，存储成本占整机物料成本已从2025年第三季度的22%升至2026年第三季度的64%。他预计，2027年售价低于1500元的智能手机将很难看到。

不过，在消费端收缩的同时，新的需求来源也在积蓄。

林美炳告诉记者，车规级存储在中国市场的增长非常强劲，ADAS（高级驾驶辅助系统）的快速渗透正推动智能座舱和自动驾驶域控制器对存储容量需求的大幅提升，中国区车规存储的同比增速预计达到70%—80%。

此外，CXL（一种支持服务器之间内存池化共享的互联技术）预计2028年前后将随着英伟达和谷歌的采用开始放量，目前各家厂商已有样品，但尚未量产。另外，高带宽闪存HBF（介于HBM和固态硬盘之间的新型存储层级）的首个行业标准规范也在8月4日由SK海力士和闪迪联合发布，预计2027年底出货。

2028年之后，如果这些新需求来源起量，有可能会部分对冲新增存储产能带来的供给宽松。

前述存储模组厂商的产品经理认为，目前存储的涨价斜率虽然在逐季收窄，但三巨头在2027年的利润规模仍然有可能超过2026年。他强调，市场现在争论的焦点集中在“周期何时见顶”，但对三巨头来说，更重要的变量可能是长协覆盖的比例能走到多高，“美光的目标是40%以上，三星已经到了60%—70%，如果最终行业的长协平均覆盖率稳定在50%以上，那么即便现货市场出现波动，原厂有一半以上的收入是锁定的，盈利的波动幅度会比以往任何一轮周期都小得多”。

频道: 金融财经

本内容来源于网络 原文链接，观点仅代表作者本人，不代表虎嗅立场。

如涉及版权问题请联系 hezuo@huxiu.com，我们将及时核实并处理。

正在改变与想要改变世界的人，都在 虎嗅APP

The honest starting position is to follow a specific checklist before looking at the model's internals. If your model is producing bad output, the ordered list of things to do begins: read the actual prompt that was sent, including everything your framework appended; check the retrieved context for the failing case; check whether the failure reproduces at temperature 0; and check whether it started at a specific deploy. Those four steps resolve the overwhelming majority of real incidents. The internals of the model are the last place to look, not the first, because they are the part that did not change — and the systematic version of that checklist is a better use of an afternoon than any technique on this page. What follows is for the residue: problems that survive that pass, and where a signal from inside the model is genuinely the cheapest way forward.

The gate: do you have the weights?

Almost everything in this cluster requires local weights. A hosted API returns tokens and, sometimes, log-probabilities. It does not return activations, and it will not. So the practical question is not “is this technique good” but “is this problem worth self-hosting an open model to investigate”. Usually it is not. Occasionally — a classifier at the centre of a product, a safety-relevant behaviour you must characterise, a model you fine-tuned and now cannot explain — it is, and the self-hosting trade-off changes when internals access is part of what you are buying. One technique survives the gate: log-probability analysis works through any API that exposes it, and it is the cheapest useful signal about a model’s internal state that exists.

What pays for itself

**Log-probabilities, on any model**

The token distribution tells you whether the model was confident or picking between near-ties. A wrong answer at probability 0.95 and a wrong answer at 0.31 are different bugs: the first is a representation problem, the second is often a prompt or retrieval problem where the right answer was in contention and lost. For any constrained output — classification, routing, extraction from a fixed set — reading the distribution over the permitted tokens gives you a usable confidence signal and an abstention threshold. That is a production feature, not an analysis: abstaining below a threshold converts a class of silent wrong answers into a handled case.

**Embedding-space diagnosis for retrieval**

When a retrieval system returns the wrong documents, the internals in question are the embeddings, and they are fully available to you. Check the similarity between the query embedding and the embedding of the document that should have won, then check the documents that actually won. This distinguishes three failures that look identical from the outside: the document was not in the index at all, it was there but scored below the cut, or it scored well and was displaced by near-duplicates. Each has a different fix, and no amount of prompt iteration separates them.

**A probe as a runtime classifier**

If you self-host, a linear probe on a mid-layer hidden state is a genuinely practical component. It costs a dot product on a tensor you already computed, it runs on the prompt before generation, and it can flag categories — this request looks like the class we route elsewhere — more cheaply than a second model call. The evaluation obligations from the probing page apply in full: a control task, a held-out split at document level, and a measured false positive rate on real traffic. A probe deployed on the strength of its training accuracy is a liability.

**Attention patterns for long-context routing bugs**

Narrow but real. If a model ignores instructions placed in the middle of a long document, the attention pattern shows the positional structure directly. This is a legitimate use because the claim being made is about routing, which is what attention reports, rather than about causation. The fix is usually moving the instruction, and having seen the pattern makes that a decision rather than a guess.

Before reaching for any of this, the request path usually has the answer. Multigrid records the model, the token counts and the latency for every request, which is what tells you whether a regression started at a specific deploy, whether it correlates with prompt length, and whether it is one model or all of them. A behaviour that appeared on a Tuesday is a deployment question, and no activation will tell you that.

What does not pay yet

**Circuit analysis of a production model**

Months of work on a narrow behaviour in a small model. There is no version of this that fits inside an incident.

**Steering vectors as a behaviour control**

Real, and almost always beaten by a system prompt plus an output check, which are auditable, portable across model versions and do not need a coefficient sweep. See the steering page for the cases where it does win.

**Sparse autoencoder features as monitoring**

The research is promising. Operationally it means training and maintaining a second model to interpret the first, and then validating that its features mean what their labels say. Watch it; do not build on it yet.

**Attribution maps as user-facing explanations**

Covered on the saliency page: the most convincing maps were the ones that failed the sanity checks. Showing users an explanation you cannot verify is worse than showing none.

If you do build it, build it small

Write the eval set first. Twenty to fifty cases that fail, with expected outputs. Without it you cannot tell whether anything you learn helps, and building it frequently solves the problem outright. Reproduce on an open model. If the bug does not reproduce on a model whose weights you have, internals are not available and the investigation ends here.

**How safety layers actually fail**

They rarely fail by missing something dramatic. They fail in one of three quieter ways: tuned so tight that users route around it, tuned so loose it is decoration, or unmeasured and therefore unchangeable. The common cause is the absence of a labelled evaluation set. Almost everything else follows from fixing that.

**The four numbers, and the one that misleads**

Every classification decision is one of four outcomes: true positive (harmful, blocked), false positive (benign, blocked), true negative (benign, allowed), false negative (harmful, allowed).

* **Precision** = TP / (TP + FP). Of what you blocked, how much deserved it. Low precision is what makes users hate the product and campaign for the bypass flag. * **Recall** = TP / (TP + FN). Of what deserved blocking, how much you caught. Low recall is what the incident review is about. * **Accuracy** = the proportion of all decisions that were correct — and it is nearly useless here. If 0.1% of your traffic is harmful, blocking nothing at all scores 99.9%. Never report accuracy for a safety layer; it is the number that makes a broken filter look excellent. * **Precision and recall trade against each other** as the threshold moves, and there is no correct point in the abstract — only a point that reflects your relative cost of the two errors. Write that ratio down explicitly. A public-facing surface with regulatory exposure and an internal drafting tool should land in obviously different places, and if your two surfaces share a threshold, at least one of them is wrong.

**Why a borrowed threshold is worthless**

Suppose a vendor reports a classifier with 95% recall and 90% precision. On traffic where 1 in 1,000 items is harmful, the precision you experience is not 90%. Of 100,000 requests, 100 are harmful, you catch 95, and the false positives come from the 99,900 benign ones. At even a 1% false-positive rate that is 999 wrongly blocked users against 95 correct blocks — a precision under 10%, from a classifier whose published numbers were good.

Nothing was dishonest in the vendor’s figure. It was measured on a balanced set, and your base rate is not balanced. This is the base-rate fallacy, and it is the reason the only threshold worth shipping is one measured on traffic shaped like yours.

**A threshold sweep you run on your own data**

Assemble a labelled set first: a few hundred items sampled from your own real traffic, labelled by two people with disagreements resolved by a third. Include the hard cases deliberately — security discussions, medical questions, fiction, quoted abuse, other languages — because the easy cases tell you nothing you did not know. Then sweep:

```typescript type Labelled = { text: string; harmful: boolean }; type Row = { threshold: number; tp: number; fp: number; tn: number; fn: number; precision: number; recall: number; blockRate: number; };

export async function sweep( set: Labelled[], score: (t: string) => Promise<number>, // your classifier, 0..1 thresholds = [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9], ): Promise<Row[]> { // Score once. Sweeping the threshold is free; re-scoring is not. const scored = await Promise.all( set.map(async (s) => ({ ...s, score: await score(s.text) })), ); return thresholds.map((threshold) => { let tp = 0, fp = 0, tn = 0, fn = 0; for (const s of scored) { const blocked = s.score >= threshold; if (blocked && s.harmful) tp++; else if (blocked && !s.harmful) fp++; else if (!blocked && !s.harmful) tn++; else fn++; } return { threshold, tp, fp, tn, fn, precision: tp + fp === 0 ? 1 : tp / (tp + fp), recall: tp + fn === 0 ? 1 : tp / (tp + fn), // The number that predicts your support load. Compare it against // the base rate: if you block 4% of traffic and 0.1% is harmful, // 39 in every 40 blocks are wrong. blockRate: (tp + fp) / scored.length, }; }); } ```

Read the table by fixing the error you cannot tolerate and taking the best available value of the other. “Recall must be at least 0.95 for this category; what is the highest precision available at that recall?” is a decision. “Which row has the best F1?” is an abdication, because F1 weights the two errors equally and you almost never do.

Then re-run the sweep on every model change, prompt change and vendor change. That regression signal is worth more than the absolute numbers, and it is the thing that stops a routine model upgrade from silently changing your safety posture.

**Building that set is where the real difficulty sits, and it is worth expecting: your labellers will disagree, sometimes on a third of the hard cases. That disagreement is data rather than noise. If two careful people cannot agree whether an item violates the policy, no classifier threshold will resolve it, and the correct fix is upstream — write the policy more precisely, or route that category to a middle action rather than to block-or-allow. Measure agreement explicitly before you measure the classifier, because a model can never score better than the consistency of the labels you graded it against.**

**Operating it**

* **Use more than two outcomes.** Allow, flag-and-log, soften, require-confirmation, block. Most items that score in the middle deserve one of the middle actions, and a binary filter forces every ambiguous case into the error you can least afford. * **Different thresholds per category and per surface.** The threshold for self-harm content and the threshold for profanity have no reason to be the same number. * **Log every decision with its score.** Without the score you cannot re-tune retrospectively, and re-scoring historical traffic is expensive. * **Give users a route.** An appeal or feedback path is both the humane choice and your best source of labelled false positives, which is the data the sweep most needs. * **Sample what you allowed.** False negatives never complain. A weekly human review of a random sample of allowed traffic is the only way they enter your numbers at all. * **Fail closed on the highest tier only.** If the classifier is unavailable, blocking everything is an outage and allowing everything is a gap. Decide per surface, in advance, and write it down.

The mechanics are straightforward: available = window - reserved_output - safety_margin

The safety margin is not superstition. Your token count is an estimate made with a client-side tokenizer against a request the provider will re-serialise, and chat templates, tool schemas and role markers all add tokens you did not write. A margin of two to three percent of the window absorbs that; being wrong in the other direction costs a failed request at the end of an expensive assembly.

Reserved output is a property of the task, not of the model. A classifier that returns one word can reserve 64 tokens, while a code generator that might emit a whole file cannot reserve less than a few thousand without occasionally truncating mid-function. Reasoning models make this sharper still, because the hidden thinking tokens come out of the same allowance — reserve for the thinking you cannot see, not just for the answer you can.

Floors, not percentages The tempting design is percentages: 20% system, 30% history, 50% retrieval. It fails immediately, because the claimants are not elastic in the same way. The system block and the tool schemas have a fixed size; they cannot be given 20% of anything. History and retrieved documents genuinely are elastic. So an allocator needs three quantities per block, not one:

* **floor** — below this the block is worthless and should be dropped entirely rather than shrunk. A retrieval block cut to 200 tokens is not a small retrieval block; it is one truncated document that will mislead. * **want** — the size the block would use if nothing competed with it. * **priority** — the order in which blocks are sacrificed when floors alone do not fit. Lower priority goes first.

The separation between shrink and drop is the part that carries most of the value. Truncating everything by a uniform fraction is the easy implementation and it degrades every block at once; a floor turns that into a decision to lose one block completely and keep the rest intact, which is almost always the better trade.

The allocator Fixed blocks are paid first, floors are paid next in priority order, and whatever is left over is shared among the elastic blocks in proportion to what they asked for. Nothing exotic — but written down, testable, and the same on every request:

```typescript type Block = { id: string; fixed?: boolean; // must be included whole or the request is invalid floor: number; // drop below this rather than shrink want: number; // size with no competition priority: number;// higher survives longer };

function allocate(blocks: Block[], available: number) { const out = new Map<string, number>(); let left = available;

// 1. Fixed blocks are not negotiable. for (const b of blocks.filter(b => b.fixed)) { out.set(b.id, b.want); left -= b.want; } if (left < 0) throw new Error("fixed context exceeds window");

// 2. Pay floors in priority order; anything unfunded is dropped. const elastic = blocks .filter(b => !b.fixed) .sort((a, b) => b.priority - a.priority); const funded: Block[] = []; for (const b of elastic) { if (left >= b.floor) { out.set(b.id, b.floor); left -= b.floor; funded.push(b); } else { out.set(b.id, 0); // dropped, not starved } }

// 3. Share the remainder in proportion to unmet demand. const demand = funded.reduce( (s, b) => s + (b.want - b.floor), 0 ); if (demand > 0 && left > 0) { for (const b of funded) { const extra = Math.floor( left * (b.want - b.floor) / demand ); out.set( b.id, Math.min(b.want, out.get(b.id)! + extra) ); } } return out; } ```

Two properties are worth stating because they are what make it worth having. It is total — every block gets a number, including zero — so downstream code never has to guess whether a block was omitted deliberately. And it is pure, so the whole of your allocation policy is covered by tests that run in milliseconds and need no model.

A worked split Assume — and every number here is an assumption, substituted for your own — a 128,000-token window, 4,000 reserved for output, and a 3,000-token safety margin. That leaves 121,000 available. Four claimants:

* Fixed blocks take 8,200, leaving 112,800. * Floors take 6,500, leaving 106,300 to share. * Unmet demand is 56,000 for history and 77,500 for retrieval, 133,500 total — so history gets an extra 106,300 × 56,000 / 133,500 ≈ 44,600 and retrieval gets ≈ 61,700.

Final split: history 48,600, retrieval 64,200. Both are under their want, so both will compact, and both know by how much before either one runs.

Now change one input and watch the policy earn its keep. Move to a 32,000-token window with the same reservations: available is 25,000, fixed takes 8,200, floors take 6,500, and only 10,300 is left to share. Nothing is dropped, but retrieval lands near 8,500 — about one and a half documents. That is the moment to notice that your retrieval step should be returning three documents rather than twelve, which is a filtering decision the allocator has just made visible.

Where a budget goes wrong * Counting the wrong string. The budget must be computed over what is actually sent — after the chat template, with tool schemas serialised exactly as the SDK will serialise them. Counting the raw text under-reports, sometimes badly. * An unbounded block that nobody declared. Tool results are the usual culprit: they arrive between requests, are appended by the framework, and never passed through allocate at all. Give them a block or they will quietly own the window. * A budget that changes on every request. If the allocation shifts by a few tokens each turn, the prompt prefix changes each turn, and every cache hit is lost. Round allocations to a coarse granularity so the prefix is stable. * Silent overflow. The failure mode of a missing budget is not an exception; it is a provider or a framework truncating from one end without telling you, which produces a model that has simply not read part of its instructions. * The reservation half of the arithmetic depends on two per-model numbers that change whenever you change model, and guessing them is how an allocator ends up tuned for a model you no longer call. The catalogue lists context length and maximum output per model so window and the ceiling on reserved_output can be read rather than assumed.