Notes from the nest · 980 posts
Engineering notes written by the people building Crowkis. Comparisons, use cases, economics, internals, security, operations, and nothing written just to rank.
Least-recently-used eviction is semantically blind. It happily discards the rare, costly answer you'll pay dearly to regenerate, and keeps the cheap trivia everyone asks. There's a smarter way.
Building meeting-notes summarizers on Dify? Add a semantic cache so similar summaries requested repeatedly stop costing full price.
Building RAG document search on the Mistral SDK? Add a semantic cache so the same questions re-running retrieval over the same corpus stop costing full price.
Building legal document assistants on LiteLLM? Add a semantic cache so the same clauses and questions across matters stop costing full price.
Building data analysis agents on Haystack? Add a semantic cache so repeated tool calls and the same analytical questions stop costing full price.
Prompt versioning and A/B testing at the data layer. A practical, Crowkis-grounded take, no hype, just what actually moves cost, latency, and safety.
Prompt versioning and A/B testing (CPROMPT), versions prompts on every write, renders variables, and runs sticky per-user A/B splits, so you roll back or test without a code deploy. Here's how Crowkis does it and why it matters for cost and safety.
Chain-of-thought costs several times more tokens than the answer it produces, and it's usually thrown away. For structurally similar problems, most of that reasoning can be reused.
Building SQL generation tools on Dify? Add a semantic cache so the same schema questions and query shapes stop costing full price.
Building internal copilots on the Mistral SDK? Add a semantic cache so employees asking overlapping questions of the same knowledge base stop costing full price.
Building education tutors on LiteLLM? Add a semantic cache so students asking the same concepts thousands of times stop costing full price.
Building HR assistants on Haystack? Add a semantic cache so the same policy questions from every employee stop costing full price.
Streaming responses from cache without breaking UX. A practical, Crowkis-grounded take, no hype, just what actually moves cost, latency, and safety.
Self-hosted RAG (CDOC), adds documents with auto-chunking and metadata, then runs filtered ANN search with optional reranking, no separate vector database. Here's how Crowkis does it and why it matters for cost and safety.
'What's 2+2?' needs a near-exact match to reuse safely. 'Give me creative ideas for X' can tolerate a loose one. A single global threshold guarantees you're too strict somewhere and too loose somewhere else.
Building devops copilots on Dify? Add a semantic cache so the same runbook and incident questions stop costing full price.
Building chatbots on the Mistral SDK? Add a semantic cache so high-volume conversational traffic that repeats constantly stop costing full price.
Building multi-agent systems on LiteLLM? Add a semantic cache so a swarm of agents asking overlapping questions stop costing full price.
Building IT helpdesk bots on Haystack? Add a semantic cache so the same tickets and fixes, endlessly stop costing full price.
The case for a Redis-compatible AI cache. A practical, Crowkis-grounded take, no hype, just what actually moves cost, latency, and safety.