Topic · 37 posts
How Crowkis is built: the Rust LSM storage engine, the in-process vector index, and the internals behind a sub-millisecond semantic cache.
Crowkis speaks RESP3, so redis-py, ioredis, and Lettuce connect unmodified. Adoption is a port change, not a rewrite, and the semantic commands sit right beside the familiar ones.
Redis is a masterpiece, for exact-match lookups. But nobody asks your app exact-match questions. Here's why we kept its wire protocol and taught the cache to understand meaning.
Finding the nearest meaning among a million cached answers, in under a millisecond, without a single external dependency. A look at the pure-Rust HNSW engine underneath Crowkis.
Pinecone, Qdrant, Weaviate, excellent tools, genuinely. But a lot of teams reach for a whole vector database to do something a meaning-aware cache already does, with less to operate.
OpenAI and Anthropic added prompt caching, and it's genuinely useful. But it only discounts the prefix you repeat verbatim. The moment the wording changes, you pay full price again.
Credit where it's due: the first semantic caches showed the world this works. Then we asked what a production-grade version would look like if you owned every layer.
A cache that forgets everything on restart isn't much of a memory. Crowkis writes durably first, so a crash, a deploy, or a reboot never costs you a single learned answer.
Why Rust for AI infrastructure. A practical, Crowkis-grounded take, no hype, just what actually moves cost, latency, and safety.
Sub-millisecond retrieval: why in-process beats a network hop. A practical, Crowkis-grounded take, no hype, just what actually moves cost, latency, and safety.
Least-recently-used eviction is semantically blind. It happily discards the rare, costly answer you'll pay dearly to regenerate, and keeps the cheap trivia everyone asks. There's a smarter way.
The case for a Redis-compatible AI cache. A practical, Crowkis-grounded take, no hype, just what actually moves cost, latency, and safety.
Bring your own embedder / reranker, swaps in any sentence-transformers MiniLM or GTE export via ONNX, so you're never locked to the default model. Here's how Crowkis does it and why it matters for cost and safety.
The most secure dependency is the one that isn't there. Crowkis ships as a single stripped binary with the model baked in, no Python, no package manager, nothing to poison at runtime.
Once meaning is a point in space, the hard part is finding the nearest point out of millions, fast. HNSW is the elegant trick that makes it feel instant, here's the intuition.
Classic caches taught the industry a durable lesson: never compute the same thing twice. LLMs just changed what 'the same thing' means, from identical bytes to identical meaning.
A cache lives in the hot path of every request. The language choice isn't aesthetic, it's the difference between predictable microseconds and mystery pauses.
Redis-compatible (RESP3), speaks RESP3 so redis-py, ioredis, and Lettuce connect unmodified across 40+ commands, adoption is a port change. Here's how Crowkis does it and why it matters for cost and safety.
Pure-Rust LSM storage engine, is a from-scratch WAL + MemTable + SSTable + bloom-filter + compaction engine in Rust, no RocksDB and no FFI. Here's how Crowkis does it and why it matters for cost and safety.
In-process HNSW vector index, keeps a custom, persistent HNSW graph in the same process as the store, so a lookup and its scoring never cross a network hop, sub-millisecond search. Here's how Crowkis does it and why it matters for cost and safety.
Group-commit WAL, batches fsyncs on a timer instead of per write, for materially higher write throughput when you enable it. Here's how Crowkis does it and why it matters for cost and safety.
i8 vector quantization, stores vectors in int8 for about 4x less memory, so more of your working set fits in RAM. Here's how Crowkis does it and why it matters for cost and safety.
347 integration tests, a smoke suite that kills the process on purpose, and a Docker image hardened before anyone asked. The receipts behind 'production-ready.'
Most semantic caches call out to a vector database. Crowkis embeds the HNSW graph in-process, and that placement decision is worth more than any algorithm tweak.
Durability isn't a checkbox, it's a sequence of writes in the right order with checksums at every step. Here's the boring machinery that makes restarts uneventful.
The fastest disk read is the one that never happens. A few bits per key let Crowkis skip files that can't contain your answer, at a 1% false-positive cost we chose on purpose.
One similarity threshold for all traffic is how caches embarrass themselves. Crowkis classifies every query into one of twelve intents, each with its own rules of reuse.
Embeddings blur exactly where caches need precision, numbers, dates, entities. Template abstraction catches what cosine similarity structurally cannot.
Chain-of-thought tokens are the most expensive ones you buy. Crowkis extracts the thought's skeleton, abstracts the specifics, and recomposes it for the next input that shares its shape.
Every sane checklist says don't write your own storage engine. We did it anyway. Here's the actual reasoning, the architecture, and the parts that were painful.
LRU evicts by recency and nothing else. But cache entries have wildly different replacement costs, and forgetting a $0.40 answer to keep a $0.0004 one is just bad accounting.
Answers age at different speeds, prices in days, math never. A single TTL knob can't express that, so Crowkis ships five policies plus version pinning and webhooks.
Every new API is a tax on adoption: clients, docs, muscle memory, tooling. RESP3 meant inheriting twenty years of all four on day one.
Crowkis serves thousands of connections through async IO, then funnels every cache decision through a single deterministic actor. Here's why that's a feature.
MCP turns Crowkis into something an AI assistant can use deliberately, check the cache, store the answer, over plain stdio, with the banner silenced so JSON-RPC stays clean.
LSM compaction is where storage engines breed complexity. Crowkis ships exactly one strategy across three levels, chosen for cache workloads, closed for configuration.
Users expect LLM answers to arrive as a typing stream. CGETSTREAM serves cached answers chunk by chunk, so a sub-millisecond hit doesn't break the interface's rhythm.
Bottom-heavy by design: the layers that hold your data get the most hostile coverage, and the smoke suite's signature move is killing the process to prove a point.