>·10 min
TSX Costs 94% More LLM Tokens Than Pug
Four markups, one tokenizer: TSX uses 94% more tokens than Pug. But the interesting cost is not where most developers expect it to be.
llmwebdeveloper-tools
>read more_
> booting the archive…
Loading
[ // tag: #tokenizer ] (3)
>·6 min
Splintr vs Gigatoken: A Fair Comparison of Rust Tokenizers
A reader said splintr beats gigatoken. After studying its benchmarks, the answer is nuanced: it wins on flexibility and latency, gigatoken on raw throughput.
rusttokenizerllm
>read more_
>decode·5 min
Gigatoken: 989x Faster Tokenization in Rust
Gigatoken is an MIT-licensed Rust tokenizer that encodes an 11.9 GB corpus at 24 GB/s — up to 989x faster than HuggingFace and 681x faster than tiktoken.
rusttokenizerllm
>read more_