> booting the archive…
we use a small set of cookies. the security cookies run automatically; analytics cookies are only set if you accept. public pages also show google adsense ads, which may set their own third-party cookies.
> booting the archive…
A reader said splintr beats gigatoken. After studying its benchmarks, the answer is nuanced: it wins on flexibility and latency, gigatoken on raw throughput.
Gigatoken is an MIT-licensed Rust tokenizer that encodes an 11.9 GB corpus at 24 GB/s — up to 989x faster than HuggingFace and 681x faster than tiktoken.