CSTALE: serve the slightly-old answer now, refresh it behind the scenes
A hard TTL turns a one-second-expired answer into a full model call. CSTALE serves the cached answer past its TTL with a stale flag, so you choose freshness versus latency per request.
Hard expiry is a blunt instrument: the moment an entry's TTL passes, the next request eats a full model call, even if the answer is one second stale and perfectly usable. CSTALE adds the stale-while-revalidate pattern that CDNs have used for years — it returns the cached answer even past its TTL, marked with a stale flag so the caller knows.
That flag is the control surface. For a dashboard number or a chat reply, a few-seconds-stale answer served instantly is the right call, and you ignore the flag. For a price or a balance, you treat the flag as a signal to kick off a refresh — serve the stale value for snappiness while the fresh one computes, or block for the fresh one when correctness outranks speed. The policy is yours, per request.
Reuse only when meaning, structure, confidence, and trust all agree.
The win is that expiry stops being a latency cliff. Instead of every TTL boundary becoming a synchronous model call for some unlucky user, the stale answer covers the gap while the cache refreshes, and the p99 that a hard TTL would spike stays flat.
The bottom line
Freshness and latency are a dial, not a switch. CSTALE hands you the dial — most answers can tolerate a moment of staleness in exchange for never paying the cold-miss tax, and the ones that can't are exactly the ones the flag lets you treat differently.