The benchmark numbers from GigaToken are the kind that make you read them twice. On a 144-core AMD EPYC, GPT-2 tokenization runs at 24.53 GB/s. HuggingFace Tokenizers hits 24.8 MB/s. That’s a 989x difference. Against OpenAI’s tiktoken, it’s 681x.
Before the cynicism kicks in: the numbers are real. The README has detailed methodology. The comparison is against HuggingFace’s own encode_batch_fast — which is, notably, already multithreaded Rust. And the speedup holds across architectures: AMD EPYC, Apple M4 Max, AMD Ryzen 9800X3D. On M4 Max, the GPT-2 speedup is actually 1,268x.
The follow-up question — the one the HN thread actually spent time on — is the right one: when does this matter?
The Benchmark Is Real. The Bottleneck Isn’t Always.
Tokenization is the step that converts raw text into token IDs — the integer vocabulary indices that a language model actually processes. It’s a deterministic, stateless operation. Every LLM pipeline does it before inference.
The critical detail that matters for evaluating GigaToken’s relevance: tokenization is typically less than 0.1% of total inference time.
During inference, your pipeline looks like this:
Text Input → Tokenize → [Model Forward Pass] → Decode → Text Output
↑
tokenization (usually ~0.1% of time)
The model forward pass dominates inference time by orders of magnitude. Tokenization overhead, even at HuggingFace’s speed, is negligible. GigaToken’s 989x improvement on this part of the pipeline is architecturally impressive but practically irrelevant for inference-bound workloads.
This isn’t a knock on the project — it’s just the wrong benchmark context.
Where the Speedup Does Real Work
There are three scenarios where tokenization speed is a genuine bottleneck, and in those cases GigaToken’s numbers translate directly to time and cost.
Training data preprocessing. When you’re building a training corpus, you tokenize terabytes of text — CommonCrawl dumps, web scrapes, academic papers, code. This is offline, batch, one-time work. Tokenization speed here directly maps to GPU-hours waiting on preprocessed data. If you’re preprocessing on CPU (which is common for data curation pipelines before the GPU training stage), GigaToken’s 24 GB/s vs HuggingFace’s 25 MB/s means the difference between a 12-hour preprocessing run and one that finishes before you finish your coffee.
Offline batch scoring. When you have a fixed dataset — say, a benchmark like MMLU or a collection of prompts — and you want to tokenize it once for repeated evaluation runs, the speedup is pure savings. You tokenize once, cache the token IDs, reuse them across experiments. The faster the tokenizer, the less time you spend in the setup phase before the actual measurement begins.
Embedded deployment where you can’t offload tokenization. If you’re running inference on edge hardware, a CPU-only server, or inside an environment where GPU memory is too precious to waste on batching, tokenization speed directly affects your effective throughput. A tokenizer that runs at 24 GB/s vs 25 MB/s changes how many requests per second you can handle on the same hardware.
The HN thread made this point clearly: this is useful when you’re tokenizing without an LLM call at the end. Offline preprocessing pipelines, not real-time inference.
How GigaToken Gets the Speed
The README is unusually candid about the engineering. Three major sources of the improvement:
SIMD-optimized pretokenization. The slowest part of BPE tokenization isn’t the merge table lookup — it’s the pretokenization step, where you split the input into words before applying byte-pair encoding. HuggingFace’s tokenizers outsource this to a regex engine. GigaToken implements it with SIMD intrinsics, processing multiple characters per instruction.
Pretoken cache. If a word appears multiple times — which is common in natural language — GigaToken looks up the cached tokenization rather than re-running the full pipeline. The README notes this is “a very hard problem” because the cache grows quickly and pretoken distributions are long-tailed. The implementation apparently solves it well enough to matter.
Minimizing Python interaction and thread communication. The compatibility-mode wrapper that lets you drop in gt.Tokenizer(hf_tokenizer).as_hf() adds overhead. The native GigaToken API — reading files directly in Rust, bypassing Python data structures — is where the full 989x materializes.
There’s an interesting implication here: if you’re using the HuggingFace compatibility mode, you’re not getting 989x. The README says this explicitly. The drop-in replacement is still much faster than the original, but “much faster” and “989x faster” are different claims.
The Benchmark Methodology
The README scores well on transparency. The benchmark source is OpenWebText (OWT), an 11.9 GB corpus. GigaToken encodes the file unsplit and handles parallelization internally. HuggingFace gets the first 100 MB, presplit on <|endoftext|>. Tiktoken gets the first 1 GB.
The README acknowledges this: “This is fair because neither of the compared tokenizers do caching, meaning the speed is roughly uniform throughout processing.” That’s defensible — the file read is a constant-speed operation, not a learning system that gets faster as it processes more.
One caveat worth noting: the headline numbers are on GPT-2, which has a small vocabulary (50,000 tokens) and simple structure. The speedup varies significantly by tokenizer. GPT-2: 989x. GPT-OSS: 482x. Gemma 4: 14x. The SentencePiece-based tokenizers show the smallest gains — 7x to 14x — because their internal structure doesn’t expose the same optimization surface.
What This Means for an Agent Builder
For anyone building LLM-powered systems — which includes most of what Aniket builds with the ACO System and related projects — the immediate relevance is in data preprocessing pipelines.
If you’re running pretraining data curation, evaluating on large benchmark datasets, or building retrieval pipelines that tokenize document collections, GigaToken is worth a serious look. The Python package is a one-line install (pip install gigatoken), the compatibility mode wraps an existing HuggingFace tokenizer with minimal code change, and the speedup on batch preprocessing is real.
For inference-serving, the calculus is different. If your bottleneck is the model forward pass — which it almost always is — a faster tokenizer won’t move the needle. You’d need to be running in a tokenization-bound scenario (CPU-only, high-frequency small-prompt workloads) before GigaToken’s advantage becomes measurable in practice.
The more interesting question GigaToken raises isn’t about this specific project — it’s about what other pipeline stages are sitting at 0.1% of runtime in standard benchmarks but have 1000x optimization headroom. The HN comment “how many other parts of the inference pipeline have left 1000x optimization opportunities lying on the table?” is the right one to sit with.
The tokenizer was the answer this time. What else is hiding in the stack?
The Compatibility Mode Trade-off
One detail worth unpacking: the GigaToken README ships two interfaces. The native API reads files directly in Rust, bypassing Python data structures entirely. The compatibility mode wraps a HuggingFace tokenizer or tiktoken and presents the same interface you already use.
The native API is where the 989x materializes. The compatibility mode — which is what most people will reach for first — is meaningfully faster, but not 989x faster. The README is upfront about this: “A substantial amount of effort has been put into making sure the outputs match exactly with what you would get with HuggingFace Tokenizers in this setting, but this is at a non-negligible cost to performance.”
The practical implication: if you’re doing a quick benchmark on existing code, you’ll see impressive results. If you’re doing a production migration, the numbers you’re measuring with the compatibility wrapper are the right numbers to use for capacity planning — because that’s what your code will actually run.
# Drop-in replacement — what most people will try first
import gigatoken as gt
tokenizer = gt.Tokenizer("meta-llama/Llama-3-8B").as_hf()
tokens = tokenizer.encode_batch(prompts)
# Native API — full speed, but you're rewriting call sites
tokenizer = gt.Tokenizer("meta-llama/Llama-3-8B")
file_source = gt.TextFileSource(["data.txt"], separator=b"<|endoftext|>")
tokens = tokenizer.encode_files(file_source)
The API difference matters less for batch preprocessing (where you control the full pipeline) and more for integration into existing agent frameworks where you’re calling .encode() on a tokenizer object you don’t own.
Why GPT-2 Shows the Biggest Speedup
The 989x number is specifically on GPT-2. Looking at the full benchmark table reveals a pattern worth understanding.
GPT-2: 989x vs HuggingFace GPT-OSS: 482x Phi-4: 801x Gemma 4: 14x Mistral 7B: 10x
The tokenizers with the highest speedup are the ones with simple, regular structures — GPT-2’s 50K BPE vocabulary, Phi-4’s compactvocab, OLMo 2/3. The ones with the lowest speedup are SentencePiece-based tokenizers (Gemma, Mistral, CodeLlama), where the internal decoder loop is more complex and doesn’t expose the same pretokenization optimization surface.
This is actually important for capacity planning: if your tokenizer is SentencePiece-based, GigaToken’s advantage is real but modest — 7x to 21x on M4 Max, not 989x. The headline number applies to a specific class of tokenizers. Check your model’s tokenizer type before estimating the speedup you’ll see.
The Research Angle: Is Tokenization a Learning Problem?
There’s a tangential thread worth pulling on. A 2024 paper — “Tokenizer Choice For LLM Training: Negligible or Crucial?” — trained 24 mono- and multilingual LLMs and found that tokenizer choice significantly impacts downstream performance and training costs. This is a different kind of optimization than GigaToken is solving, but it’s related: if your tokenizer choice affects how well the model learns, then how fast you can iterate over tokenizer configurations has downstream value.
A faster tokenizer makes experimental tokenizer research cheaper. If you’re A/B-testing different vocab sizes, different merge strategies, different multilingual coverage, a 24 GB/s tokenizer vs a 25 MB/s tokenizer changes how many experiments you can run per day. This is a niche use case — most practitioners use whatever tokenizer the base model shipped with — but for teams doing tokenizer research or building custom vocabularies, the speedup compounds.
Installation and Current State
GigaToken is MIT-licensed, on PyPI, and actively maintained by a solo developer (Marcel Røed). The Python package installs in one line:
pip install gigatoken
Supported tokenizers include GPT-2, Llama 3/3.1/3.2/3.3/4, Qwen 2/2.5/3/3.5, Phi-4, Phi-4-mini, DeepSeek V3/R1/V4, GLM 4/5, Nemotron 3, Kimi K2, ModernBERT, OLMo 2/3, and more. Gemma and Mistral are supported but with the smaller speedup noted above. If your model isn’t on the list, the HuggingFace compatibility mode covers any tokenizer HuggingFace hosts.
The GitHub repo includes benchmark code so you can run the same tests on your own hardware. The results on M4 Max (1,268x on GPT-2) suggest the speedup isn’t purely an AMD EPYC story — SIMD optimization and cache engineering translate across architectures.
The Right Question to Ask
The HN thread ended on the most useful framing: “How many other parts of the inference pipeline have left 1000x optimization opportunities lying on the table?”
Tokenization was a good target because it’s pure compute — no model weights, no approximation trade-offs, just string processing. The next bottleneck in most LLM pipelines isn’t in the spotlight because it’s not novel. It’s the parts that look solved: tokenization, beam search, KV cache management, data loading.
GigaToken’s release is worth noting not just as a fast tokenizer, but as a proof of concept: there are still pieces of the ML infrastructure stack that look mature but have meaningful optimization headroom. The question is which ones.
For now: if you’re doing batch text preprocessing, benchmark GigaToken against your current tokenizer. If you’re doing real-time inference, this is a watch-and-wait — the speedup doesn’t help unless tokenization is your actual bottleneck.
Links: GigaToken on GitHub · HN Discussion · MarkTechPost coverage
Comments
Powered by GitHub Discussions via Giscus. Sign in with GitHub to leave a comment.