Semantic + structural cache matching: how it works and when to use it
Semantic + structural cache matching, matches questions by meaning (HNSW vectors) and by structure (12 intent-class templates), so paraphrases hit but a changed number, entity, or negation does not. Here's how Crowkis does it and why it matters for cost and safety.
Production LLM traffic is deeply repetitive, and repetition is exactly what a bill is made of. Semantic + structural cache matching is how Crowkis matches questions by meaning (HNSW vectors) and by structure (12 intent-class templates), so paraphrases hit but a changed number, entity, or negation does not.
How it works
Crowkis matches questions by meaning (HNSW vectors) and by structure (12 intent-class templates), so paraphrases hit but a changed number, entity, or negation does not. It runs inside one Redis-compatible engine, so it composes with semantic caching, agent memory, and the other intelligence layers instead of being a separate service you wire together.
CSET "how do refunds work?" "5-7 business days." CGET "refund timeline?" # a paraphrase, still a safe hit
Why it matters
Repetitive LLM workloads are where the money is, and semantic caching can cut costs up to 60-70% on repetitive workloads. Semantic + structural cache matching is part of what makes that reuse safe rather than reckless, the difference between a cache you trust in production and one you audit after every incident. Runs self-hosted with zero egress, nothing leaves your machine.
Infrastructure earns the critical path one boring, verifiable feature at a time.