lemmas: Reliability Primitives for Any LLM API

What it is

lemmas is a small, open-source Python library I built. It sits between your code and whatever LLM provider you're already calling, OpenAI, Anthropic, Gemini, Groq, vLLM, llama.cpp, or anything OpenAI-compatible, and adds reliability techniques pulled straight from published AI research.

It's not a framework, and it's not trying to get you to switch providers, clients, or how your app is built. Each primitive (that's just the name for one of these small reliability building blocks) takes a plain callable, a function that sends messages to a model and gets text back, and hands you a result object with everything it learned along the way: a baseline answer next to a revised one, vote counts behind a plurality answer, a confidence score, whichever call finished first. You keep your own client code. You just wrap the one function that talks to the model.

Here's how I put it in the README: "no new framework, no SDK lock-in, just small modules that wrap whatever provider you already use." That's the whole pitch.

What it does, the four primitives

The README frames the core of the library as four primitives, each aimed at a different way an LLM can fail once you put it in front of real traffic.

In the README's own words, together these four cover "factuality at inference time (CoVe), reasoning robustness (self-consistency), observability of behavior change (drift), and tail-latency control (hedged execution)." Each module runs about 150 lines. The whole library comes in under 1,000 lines of code with no dependencies beyond numpy.

The repo actually ships three more primitives beyond that original four, and they compose with the first four: best_of_n (sample N answers, score each one with a scorer function, that could be an LLM acting as judge, a length check, keyword matching, or your own reward model, then return the highest scorer), reflexion (an iterative try, critique, retry loop where a critic's feedback gets fed back into the next attempt, from Shinn et al. 2023), and debate (multi-agent debate, where N agents each draft an answer, then revise after seeing each other's drafts, over R rounds, from Du, Li & Mordatch 2023). Every primitive, including these three, has an async twin under lemmas.asyncio that runs the N sample calls in parallel with asyncio.gather, so what would be N times the wait becomes roughly one wait.

Why this might matter

I hold two views on this at the same time, and I think both are honest.

The modest read: lemmas packages four separately published, well-established reliability techniques, Chain-of-Verification (Meta, 2023), self-consistency sampling (2022), a practical drift-detection utility, and the hedged-execution pattern from Google's "Tail at Scale" paper (2013), behind one consistent interface that doesn't care which provider you use. I say plainly in the README why I built this: platforms like LangChain, LlamaIndex, and LiteLLM give you routing and abstractions, not these inference-time reliability methods, so "you end up reimplementing CoVe and self-consistency by hand in every project." My own framing there is direct: these are "the ones I've reached for repeatedly." A small, auditable, dependency-light, MIT-licensed library that saves you from redoing that work every time is a modest but genuinely useful thing on its own.

The bigger bet: as more products call LLM APIs directly instead of pulling in a full agent framework, a thin reliability layer that isn't tied to any one provider could turn into one of those small, focused utilities that outlasts the bigger frameworks around it. Taking callables instead of clients, and making the primitives composable, the README shows hedging between two providers and then verifying the winner's output with CoVe, or running self-consistency while feeding the same prompt to a drift detector, is exactly the kind of design that lets a library slot into a lot of different stacks without asking anyone to buy into a platform.

Honest middle ground: the README has no adoption numbers, no download counts, no count of dependent repos. So this is a read of the design, not proof of where it's headed.

What it isn't

Trust matters more to me than hype, so here's what lemmas doesn't do, in the project's own words and by honest reading of the README:

• No router. The README says it straight: "Use LiteLLM, Portkey, or your own."
• No observability backend. Lemmas hands you result objects; you pipe them into Langfuse, Helicone, Datadog, or just print them. There's no dashboard.
• No retrieval or RAG. Use LlamaIndex, LangChain, or a real vector database.
• No agent loop. Use whatever agent framework you already have.
• No streaming. All primitives work on complete responses. CoVe needs the full baseline answer, self-consistency needs full samples, hedged execution returns the first complete response, and drift detection works per prompt.
• Not on PyPI yet, as of this writing. The README's own install instructions read "pip install lemmas  # once published to PyPI," with a pip install git+https://... fallback until then.
• Documentation depth isn't even across all seven primitives. Four of them (CoVe, self-consistency, DriftDetector, race) get full worked examples and their own walkthrough section. best_of_n and reflexion also get worked examples. debate shows up in the module table with a one-line description but doesn't have its own worked-example section yet. I'd rather you know the docs are uneven than have you find out by hitting a wall.
• No benchmark suite for lemmas itself. The numbers I cite for effectiveness, like self-consistency's "10-20 points" improvement, come from the original academic papers these primitives wrap, not from an eval suite this repo runs and reports on its own. The test suite here checks that the primitives' logic is correct, not how much accuracy they buy you.

These are choices I made on purpose, and I say so directly in the README: "Lemmas is a small library, not a framework."

How it works, in plain English

Each of the four core primitives follows a mechanism the README lays out explicitly. Nothing here goes beyond what's already disclosed there.

CoVe, Chain-of-Verification

Four steps. Baseline: the model writes a first answer. Plan: the model comes up with N verification questions about the kinds of claims a good answer should contain, without seeing its own baseline first, so the questions don't get biased by what it already wrote. Execute: each question gets answered on its own, so a shaky claim can't verify itself. Revise: the model rewrites the baseline using the question and answer pairs, fixing or cutting anything its own verification contradicted. This costs N+2 model calls. Per the README, it wins on TriviaQA-style questions, WikiData lists, biographies, and multi-fact questions, and it doesn't help on math, code, or single-fact lookups.

Self-consistency, sample and vote

Sample N completions at a temperature above zero (temperature is the setting that controls how much randomness the model uses when it generates text), pull an answer out of each one using one of four extractor strategies (last_line, last_number, a custom regex, or similarity, which embeds all N samples and returns whichever one sits closest to the semantic center of the group), then return whichever answer got the most votes, along with a confidence score and the vote counts. This costs N model calls. The original paper recommends N between 20 and 40 for hard reasoning benchmarks, though the README notes that N=5 is enough for most everyday tasks.

DriftDetector, rolling centroid and z-score

DriftDetector keeps a unit-norm centroid, a running average of recent prompt embeddings, normalized so its length stays at 1, for each named bucket you give it. An embedding is just a list of numbers that captures the meaning of a piece of text, so two prompts that mean similar things end up with similar numbers. On every new prompt, it measures the cosine distance between that prompt and the bucket's centroid (cosine distance is a way of measuring how different two pieces of text are in meaning, not word for word), then updates the centroid and the variance using an exponential moving average, a running average that weighs recent data more heavily than old data. That produces a z-score, a standard statistics term for how many standard deviations away from normal something sits. Drift gets flagged when the absolute value of that z-score crosses a threshold and the bucket has cleared a warmup period, so a brand-new bucket doesn't trigger false alarms on day one. State lives in memory by default, or you can persist it across processes with a persist_fn/load_fn pair, SQLite, Redis, whatever you're already running.

race, hedged execution

Runs all N callables at once inside a ThreadPoolExecutor (Python's built-in thread pool), returns whichever one succeeds first, cancels the rest, and logs the losers so you can look at them later. A failure in one callable doesn't kill the whole race unless you set fail_fast=True. This one is generic, not built for LLMs specifically, the README suggests using it to race two embedding providers, two web fetches, or two database shards against each other. In practice, because the losing calls get cancelled mid-flight, the real cost runs closer to 1.2 to 1.5 times the compute rather than the full N times, in exchange for a real cut in your worst-case latency.

The research behind this

None of these techniques are mine. lemmas just wraps them behind one interface. If you want to go read the actual papers, here's where each primitive comes from, straight from the README's citations:

• Dhuliawala et al. (2023). Chain-of-Verification Reduces Hallucination in Large Language Models. arxiv.org/abs/2309.11495, the paper behind cove.
• Wang et al. (2022). Self-Consistency Improves Chain of Thought Reasoning in Language Models. arxiv.org/abs/2203.11171, the paper behind self_consistency.
• Shinn et al. (2023). arxiv.org/abs/2303.11366, the paper behind reflexion.
• Du, Li & Mordatch (2023). arxiv.org/abs/2305.14325, the multi-agent debate paper behind debate.
• Dean & Barroso (2013). The Tail at Scale. Communications of the ACM. research.google/pubs/the-tail-at-scale, the paper behind race.

DriftDetector isn't from a single paper, it's a rolling embedding centroid combined with Welford's method for tracking variance online, a well-known statistics technique rather than a specific citation in the README.

See the code, try it

The repo is public on GitHub, MIT licensed, with a CI badge on the README pointing at a GitHub Actions workflow. Full docs, beyond what's in the README, live at NORTHTEKDevs.github.io/lemmas. The test suite runs offline: pytest tests/ completes in about 6 seconds using deterministic stub CompleteFn/EmbedFn callables, with no API keys or network calls required.

git clone https://github.com/NORTHTEKDevs/lemmas.git
cd lemmas
pip install -e .[dev]
pytest tests/          # ~6s, no network calls
ruff check .

There's also a small command-line tool if you want to try it without writing any code, including an offline mode backed by a deterministic echo stub:

# Offline (no API key needed):
python -m lemmas cove "Who painted the Mona Lisa?"
python -m lemmas self_consistency "What is 2+2?" --n 3

# With a real provider:
OPENAI_API_KEY=sk-... python -m lemmas cove "Where was Marie Curie born?"

Built by Kristian Baer / Northtek. Open source, MIT license. Citation format is in the README.