Gigatoken: 989x Faster Tokenization in Rust

Most of us never think about tokenization. It is the quiet step that turns text into numbers at the start of every LLM call — and, more often than anyone admits, the reason a data pipeline crawls. Gigatoken, an MIT-licensed tokenizer written in Rust with Python bindings, wants you to think about it. Because it encodes plain text at gigabytes per second. Yes, gigabytes, not megabytes.

The project's own benchmarks on an 11.9 GB OpenWebText corpus show GPT-2 tokenization on a dual-socket AMD EPYC 9565 (144 cores) running 24.53 GB/s. Compare that with HuggingFace's tokenizers at 24.8 MB/s and OpenAI's tiktoken at 36.0 MB/s — both already multithreaded Rust. That is 989x faster than one, 681x faster than the other. The speedup holds across machines and CPU architectures:

The tech itself is open source on GitHub under the MIT license with Python bindings on PyPI (pip install gigatoken).

Why tokenization became the bottleneck

Tokenizers looked like a solved problem. We bundle them, cache their vocab, and never profile them — model inference gets the attention while the encoder quietly eats multi-TB datasets. HuggingFace's tokenizers and tiktoken are already Rust and already parallel. Gigatoken instead optimizes the two places everybody else leaves alone: the regex pre-tokenization pass and the merge-loop parallelism.

Every mainstream tokenizer pre-tokenizes text with a large regex that splits it into "word-like" segments before the BPE merge step. That regex is a genuine bottleneck — the single slowest part of the encoder. Gigatoken replaces it with a hand-written implementation of the exact same method, then pairs it with concurrent data structures so more of the pipeline runs on more cores at once.

Two ways to use it

Gigatoken has two surfaces. The native API is where the headline numbers come from — it reads raw bytes directly and can take a whole file as one parallel batch:

import gigatoken as gt

# native API — GB/s territory
encoder = gt.Tokenizer("gpt2")
tokens = encoder.encode_batch([raw_bytes], parallel=True)

# compatibility mode: wrap an existing encoder, identical output
compatible = gt.Tokenizer(some_tiktoken_encoding).as_tiktoken()
ids = compatible.encode("hello, operator", allowed_special="all")

Compatibility mode wraps an existing HuggingFace or tiktoken encoder and promises exact token parity. It costs speed — about 200-300x instead of 1000x — because the Python layer still pays list and string-conversion overhead. But it means a drop-in swap on code you already run. Independent replications confirm the trend too: on KrabArena, a 4-vCPU Xeon VM reached 277.8 MB/s, roughly 26x past tiktoken and 83x past HuggingFace, with output validated over 35,356 documents.

Where the numbers are more modest

Now the honest caveats. Gigatoken is specialized for BPE. SentencePiece-based vocabularies only get 7-22x in the same benchmark, and WordPiece is not supported at all. The comparison is also not strictly apples-to-apples: the baselines are measured on a pre-split slice, while Gigatoken reads the whole file — which is exactly how you would want it in production. And for single-document, latency-sensitive serving, a serving-optimized tokenizer can beat it per call. Gigatoken shines on offline, multi-GB, throughput-bound jobs, not on a 50-token chat request.

Should you switch?

Two situations make me say yes today. First, you are processing a large corpus — pre-training data, semantic-index scanners, batch embeddings. Second, your pipeline is CPU-bound on tokenization and you cannot scale your cloud budget any further. Installing it is one command:

pip install gigatoken

If you serve short prompts and need exact HuggingFace behavior per request, keep your current library — Gigatoken is not for you yet. But it has reset expectations about what a step all of us considered solved can actually do. It quietly shipped to Hacker News and GitHub in July 2026, written by a Stanford PhD student. If you process text at scale, it is worth a look while the wave is early — and, like we said in our piece on AI writing pipelines, a fast tool only stays honest when a human review step is still in the loop.