Notes from the nest · 980 posts
Engineering notes written by the people building Crowkis. Comparisons, use cases, economics, internals, security, operations, and nothing written just to rank.
A vector database is built for large-scale retrieval. A semantic cache is built for safe answer reuse. Using one for the other's job is where teams get burned.
Read it →
A plain key-value cache misses the moment a prompt is reworded, and a raw vector cache can serve the wrong answer. A semantic cache understands meaning and structure, and only reuses when it's safe.
The expensive part of a hard answer is the thinking. Crowkis stores reasoning as a reusable step graph and replays it for the next question that shares its shape, at a fraction of the token cost.
The cheapest token is the one you never spend twice. Here's the simple math behind semantic caching, and where the savings actually come from.
If your corpus fits a cache, you don't need a separate vector database to do retrieval. CDOC adds documents with auto-chunking, filtered search, and reranking, all local.
Attackers disguise injections with odd spacing and character swaps. CGUARD normalizes the disguise first, then scans, so the trick that beats a naive filter doesn't beat this.
crowsight is a small, offline embedding model that ships inside Crowkis. We didn't trust it on faith, so we made it compete with the biggest embedding APIs on the one job a semantic cache actually needs. Here's what happened.
The Crowkis binary doubles as an MCP server, so Claude Desktop, Claude Code, and any MCP-capable agent can check the cache before calling the model and store what they compute.
Point your existing OpenAI client at Crowkis and change nothing else. Repeated questions are served from cache with no upstream call, and you get retries and routing for free.
Most LLM bills are quietly full of duplicates, the same question, reworded, billed at full price every time. Semantic caching is how you stop paying for an answer you already have.
Embeddings usually mean an API key and a per-token bill. CEMBED turns text into vectors using the bundled local model, for free, with nothing leaving your machine.
Crowkis speaks RESP3, so redis-py, ioredis, and Lettuce connect unmodified. Adoption is a port change, not a rewrite, and the semantic commands sit right beside the familiar ones.
Redis is a masterpiece, for exact-match lookups. But nobody asks your app exact-match questions. Here's why we kept its wire protocol and taught the cache to understand meaning.
Most agents forget the moment a session ends. Real memory consolidates contradictions, blends relevance with recency, and can even tell you what it believed at a past point in time.
If you're evaluating semantic caches, similarity is the easy part. The differences that matter in production are safety, isolation, confidence, and cost control.
Finding the nearest meaning among a million cached answers, in under a millisecond, without a single external dependency. A look at the pure-Rust HNSW engine underneath Crowkis.
Building healthcare Q&A assistants on Dify? Add a semantic cache so recurring policy and triage questions stop costing full price.
Building code review bots on the Gemini SDK? Add a semantic cache so the same review patterns across pull requests stop costing full price.
Building research assistants on the Vercel AI SDK? Add a semantic cache so overlapping literature and summary questions stop costing full price.
Building API documentation bots on AutoGen? Add a semantic cache so the same endpoint questions from every developer stop costing full price.