Skip to content

token-estimate

Estimate LLM token counts from text without loading a tokenizer.

Real tokenizers (BPE, SentencePiece) need a multi-megabyte vocabulary and a non-trivial dependency tree. That is the right tool when you need an exact count, and the wrong tool when you only need to answer "will this roughly fit in the context window" — the common case in CLI tooling, log processing, and pre-flight budget checks.

token-estimate is a pure-Rust, zero-dependency crate that gives a fast estimate and is explicit about how wrong it can be. Companion reference material lives at ClaudHQ.

Accuracy, stated honestly

For ordinary English prose the character-based estimate typically lands within about 10-15% of a BPE tokenizer. It degrades in predictable ways:

Input Behaviour Mitigation
Code and markup Tokenizes denser than prose, so the estimate runs low Use Profile::Code
Non-Latin scripts Often one token per character; a chars/4 heuristic underestimates badly Non-ASCII density is detected and the ratio blended toward 1.0 — treat CJK figures as a rough floor
Long whitespace or punctuation runs Compress well, so the estimate runs high Whitespace runs are collapsed before counting

This is for budgeting, not billing

If you need an exact count, use a real tokenizer. Do not use these figures to reconcile an invoice.

Install

[dependencies]
token-estimate = "0.1"

Quick example

use token_estimate::{estimate, estimate_auto, fits, Profile};

let n = estimate("The quick brown fox jumps over the lazy dog.", Profile::Prose);
assert!((8..=14).contains(&n));

// Code is denser: the same text yields more tokens.
let code = "fn main() { let x: Vec<u8> = vec![1,2,3]; }";
assert!(estimate(code, Profile::Code) > estimate(code, Profile::Prose));

// Profile inferred from punctuation density.
let _ = estimate_auto("if (a == b) { c[d] = e; }");
assert!(fits("short input", 100, Profile::Prose));