Splintr vs Gigatoken: A Fair Comparison of Rust Tokenizers

After we published our piece on Gigatoken, a reader commented that splintr is better. Not "splintr is also fast" — better. That is a strong claim against a library that encodes an 11.9 GB corpus at 24 GB/s. But the commenter turned out to be partly right, and the reasons are more interesting than the headline.

I spent a while with splintr's README, benchmark harness, and differential-testing setup. Here is the honest comparison, not a coronation.

What splintr actually is

splintr is an MIT-licensed Rust tokenizer with Python bindings (pip install splintr-rs), written by Farhan. Its pitch is not "fastest at one thing" — it is one handle over every format. Four tokenization backends sit behind a single AnyTokenizer type:

• byte-level BPE — GPT-2, Llama 3, Qwen, DeepSeek, Mistral v3
• SentencePiece BPE — Mistral v1/v2 (merge-by-rank)
• Unigram — T5, Gemma, Albert (true Viterbi, not greedy)
• WordPiece — BERT, DistilBERT, Electra

and they load from four sources: 18 bundled vocabularies by name (cl100k_base, o200k_base, gpt-oss, llama3, llama2, codellama, phi4, olmo2, modernbert, qwen3, glm4, kimi_k2/k3, deepseek_v3, mistral_v1/v2/v3, whisper), any HuggingFace tokenizer.json, a raw .tiktoken file, or a GGUF vocabulary. That last one is the killer feature Gigatoken simply does not have — GGUF vocabularies are exactly what llama.cpp-style runtimes ship.

The code is identical no matter which vocabulary loads, which is the real selling point:

from splintr import Tokenizer

tok = Tokenizer.from_pretrained("qwen3")          # bundled, no file, no network
tok2 = splintr.from_json("tokenizer.json")        # any HuggingFace model
tok3 = Tokenizer("vocab.tiktoken", PATTERN)       # a bare rank file

tokens = tok.encode("Hello, world!")
batch = tok.encode_batch(["Hello, world!", "How are you?"])

The benchmark picture, honestly

Splintr's own measured run (AMD Ryzen 9 5900X, cl100k_base, splintr 0.19.1 vs tiktoken 0.13.0 and HuggingFace tokenizers 0.23.1 — pinned in its README) shows where it wins:

So against tiktoken, splintr claims roughly 18x on batches and ~10x on single texts — the same rough class of "Rust is fast" numbers Gigatoken quotes against the same baselines.

Now the part the commenter would have been talking about: splintr's own README benchmarks directly against Gigatoken, same tokenizer.json, no loader asymmetry:

• Vocabulary load: splintr is 2-4x faster (bundled vocabularies are packed binary, borrowed rather than copied).
• Single text: splintr is ~1.5x faster on x86-64; a tie on Apple Silicon.
• Batch: a toss-up — either engine can win by about ±20%, flipping by machine, vocabulary, and output form.

That is a dramatically more modest claim than the "989x" headline Gigatoken prints. And it is the crux of the whole debate.

The two numbers measure different jobs

Gigatoken's 24 GB/s is whole-file, un-split encoding on a 144-core EPYC server (its published benchmark: 11.9 GB owt_train.txt on a dual-socket AMD EPYC 9565). It reads an entire multi-gigabyte corpus as one blob, chunks it internally, and saturates every core. That is the correct metric for one specific job: offline preprocessing of a giant pretraining corpus on a big machine.

Splintr's numbers are batch encoding on a 12-core desktop — hundreds or thousands of texts. That is the metric that matches what most people actually do: tokenizing RAG chunks, counting tokens for billing, encoding prompts, preprocessing a search index.

Neither benchmark is wrong. They are measuring different regimes, and the "winner" flips depending on which regime you live in.

Where splintr genuinely wins

Beyond the single-text and load-time edges, splintr has real, structural advantages:

• Four backends, one API. Gigatoken is BPE-first with weaker SentencePiece support (7-22x there) and no WordPiece. Splintr does BPE, Unigram and WordPiece all correctly — and will load whatever tokenizer.json you hand it.
• GGUF vocabularies. Gigatoken has no answer for a llama.cpp-style vocabulary. Splintr loads one via from_gguf_vocab.
• Correctness as a feature. Splintr is fuzzed id-for-id against tiktoken, HuggingFace tokenizers, and sentencepiece — and its benchmark harness refuses to report any timing until every engine produces identical ids. Gigatoken also validates output, but splintr makes differential correctness the default contract, not a per-run check.
• 54 agent tokens. ChatML, thinking, ReAct, tool-calling and RAG citation markers appended to every bundled vocabulary — a real convenience for agent work Gigatoken doesn't ship.
• Streaming decoder with proper UTF-8 boundary handling, for live LLM output.
• Smart parallelization. Sequential for single texts under ~1 MB (which is where latency lives), Rayon across texts for batches, and encode_rayon only for the rare >1 MB single text. Gigatoken's default of parallelizing everything is better for giant files but not for a 50-token chat prompt.

It also uses techniques worth respecting: a pure-Rust JIT/SIMD regex engine (regexr), Aho-Corasick for special tokens instead of regex alternation, a linked-list BPE that avoids O(N²) on pathological inputs, FxHashMap, and an LRU cache.

Where Gigatoken still wins

Raw throughput on massive corpora. On the EPYC 9565, Gigatoken holds 24.53 GB/s on GPT-2; splintr's batch figures are in the low hundreds of MB/s on a desktop. If you are encoding an 11.9 GB OpenWebText snapshot and you own the big box, Gigatoken is in a different league — and its custom pre-tokenization implementation (replacing the regex that bottlenecks every other encoder) is a genuinely clever systems result.

Gigatoken is also the more specialized tool for BPE-only shops: one vocabulary family, no extra formats, brutal speed.

Verdict: the commenter was partly right

The commenter was right in the way that matters for most readers: for ordinary workloads — RAG preprocessing, prompt encoding, token counting, mixed model families, GGUF runtimes — splintr is the better tool. It is faster on single texts, loads vocabularies 2-4x faster, covers every tokenization format in existence, and is verified id-for-id against the reference implementations.

They were wrong to frame it as a knockout. Gigatoken remains the throughput king for the narrow-but-important job of multi-GB corpus preprocessing on many-core servers. Splintr's own README says the batch comparison is "a toss-up, ±20% each way" — that is not the language of a library claiming a knockout, and both projects are honest about it.

If I were building a tokenization layer today, I would reach for splintr as the default and keep gigatoken in the back pocket for the batch-crushing jobs. The real lesson from this thread is the same one we covered in our piece on AI writing pipelines: read the benchmarks, check the regime they were measured in, and decide for yourself.

Sources

• ml-rust: ml-rust/splintr
• ml-rust: Bundled vocabularies
• ml-rust: Benchmark methodology
• marcelroed: marcelroed/gigatoken
• PyPI: gigatoken v0.10.0
• PyPI: splintr-rs