Notes from the nest · 980 posts
Engineering notes written by the people building Crowkis. Comparisons, use cases, economics, internals, security, operations, and nothing written just to rank.
Once meaning is a point in space, the hard part is finding the nearest point out of millions, fast. HNSW is the elegant trick that makes it feel instant, here's the intuition.
Building chatbots on Rig (Rust)? Add a semantic cache so high-volume conversational traffic that repeats constantly stop costing full price.
Building multi-agent systems on the Mistral SDK? Add a semantic cache so a swarm of agents asking overlapping questions stop costing full price.
Building IT helpdesk bots on LiteLLM? Add a semantic cache so the same tickets and fixes, endlessly stop costing full price.
Building meeting-notes summarizers on Haystack? Add a semantic cache so similar summaries requested repeatedly stop costing full price.
Building RAG document search on LangChain? Add a semantic cache so the same questions re-running retrieval over the same corpus stop costing full price.
OpenAI-compatible AI gateway, proxies /v1/chat/completions with a semantic cache in front, so you point your client's base URL at Crowkis and change nothing else. Here's how Crowkis does it and why it matters for cost and safety.
CGET finds a cached answer by meaning, not exact bytes, and can return the confidence behind the hit so you decide whether to trust it.
The Python SDK wraps the semantic cache in an ergonomic client, get-or-compute, streaming, tenants, models. Here's the three-line version and the production version.
Retrieval-augmented generation stuffs context into every prompt to make answers better. It also makes every repeated question dramatically more expensive, and repeated questions are most of them.
Building AI search on Rig (Rust)? Add a semantic cache so popular queries hit again and again stop costing full price.
Building voice assistants on the Mistral SDK? Add a semantic cache so latency-sensitive, repetitive spoken queries stop costing full price.
Building sales enablement tools on LiteLLM? Add a semantic cache so reps asking the same product questions stop costing full price.
Building SQL generation tools on Haystack? Add a semantic cache so the same schema questions and query shapes stop costing full price.
Building internal copilots on LangChain? Add a semantic cache so employees asking overlapping questions of the same knowledge base stop costing full price.
Multi-provider routing and fallback, load-balances and fails over across providers on error class, with retries using exponential backoff and jitter. Here's how Crowkis does it and why it matters for cost and safety.
The model's output is the other trust boundary. COUTCHECK scans responses for PII leakage and toxicity, and optionally validates JSON, returning a structured verdict you can act on.
CSIM returns the semantic similarity between two strings, the primitive behind every hit decision, exposed so you can calibrate thresholds.
Coding agents re-read the same schema, re-derive the same conventions, and re-ask the same architecture questions on every run. A shared memory turns that repeated context into a one-time cost.
Building ecommerce assistants on Rig (Rust)? Add a semantic cache so the same product and policy questions across shoppers stop costing full price.