Gigatoken: 989x Faster Tokenization in Rust

Gigatoken: 989x Faster Tokenization in Rust

Most of us never think about tokenization. It is the quiet step that turns text into numbers at the start of every LLM call — and, more often than anyone admits, the reason a data pipeline crawls. Gigatoken, an MIT-licensed tokenizer written in Rust with Python bindings, wants you to think about it. Because it encodes plain text at gigabytes per second. Yes, gigabytes, not megabytes.

The project's own benchmarks on an 11.9 GB OpenWebText corpus show GPT-2 tokenization on a dual-socket AMD EPYC 9565 (144 cores) running 24.53 GB/s. Compare that with HuggingFace's tokenizers at 24.8 MB/s and OpenAI's tiktoken at 36.0 MB/s — both already multithreaded Rust. That is 989x faster than one, 681x faster than the other. The speedup holds across machines and CPU architectures:

The tech itself is open source on GitHub under the MIT license with Python bindings on PyPI (pip install gigatoken).

Why tokenization became the bottleneck

Tokenizers looked like a solved problem. We bundle them, cache their vocab, and never profile them — model inference gets the attention while the encoder quietly eats multi-TB datasets. HuggingFace's tokenizers and tiktoken are already Rust and already parallel. Gigatoken instead optimizes the two places everybody else leaves alone: the regex pre-tokenization pass and the merge-loop parallelism — which is also why token counts show up on your inference bill.

Every mainstream tokenizer pre-tokenizes text with a large regex that splits it into "word-like" segments before the BPE merge step. That regex is a genuine bottleneck — the single slowest part of the encoder. Gigatoken replaces it with a hand-written equivalent, then pairs it with concurrent data structures so more of the pipeline runs on more cores at once.

Two ways to use it

Gigatoken has two surfaces. The native API is where the headline numbers come from — it reads raw bytes directly and can take a whole file as one parallel batch:

import gigatoken as gt

# native API — GB/s territory
encoder = gt.Tokenizer("gpt2")
tokens = encoder.encode_batch([raw_bytes], parallel=True)

# compatibility mode: wrap an existing encoder, identical output
compatible = gt.Tokenizer(some_tiktoken_encoding).as_tiktoken()
ids = compatible.encode("hello, operator", allowed_special="all")

Compatibility mode wraps an existing HuggingFace or tiktoken encoder and claims exact token parity. Per the project's own numbers it costs speed — about 200-300x instead of 1000x — because the Python layer still pays list and string-conversion overhead. But it means a drop-in swap on code you already run. One caution that applies to every third-party replication figure you see quoted for this project: re-check it against the repo's current benchmark page before repeating it — early numbers for fast-moving repos rot within weeks.

Where the numbers are more modest

Now the honest caveats. Gigatoken is specialized for BPE. SentencePiece-based vocabularies only get 7-22x in the same benchmark, and WordPiece is not supported at all. The comparison is also not strictly apples-to-apples: the baselines are measured on a pre-split slice, while Gigatoken reads the whole file — which is exactly how you would want it in production. And for single-document, latency-sensitive serving, a serving-optimized tokenizer can beat it per call. Gigatoken shines on offline, multi-GB, throughput-bound jobs, not on a 50-token chat request — how it compares against its closest rival is the natural next read.

How to Read the 989x

The headline number is a throughput ratio on many-core hardware, not a universal speedup. Ratios differ across the machines in the project's own table precisely because the gain is core-count-shaped: more cores, more of the pipeline running at once, bigger multiple. On a small VM the same code shows a far smaller advantage. Read the 989x as "this is what the parallelism buys you when cores are plentiful," not "everything tokenizes a thousand times faster now."

Throughput Is Not Latency

Batch and interactive workloads want different things. Offline corpus processing cares about megabytes per second across gigabytes; a chat request cares about milliseconds for one short prompt. Gigatoken wins the first game by construction — concurrent structures over a whole file — and explicitly does not promise the second, where per-call overhead dominates. Match the tool to the game before quoting the headline number.

Should you switch?

Two situations make me say yes today. First, you are processing a large corpus — pre-training data, semantic-index scanners, batch embeddings. Second, your pipeline is CPU-bound on tokenization and you cannot scale your cloud budget any further. Installing it is one command:

pip install gigatoken

If you serve short prompts and need exact HuggingFace behavior per request, keep your current library — Gigatoken is not for you yet. But it has reset expectations about what a step all of us considered solved can actually do. It shipped quietly on GitHub, and if you process text at scale, it is worth a look while the wave is early — and, like we said in our piece on AI writing pipelines, a fast tool only stays honest when a human review step is still in the loop.

Sources

• GitHub: Gigatoken