Packages crates.io tokenizers Package · crates.io
tokenizers crates.io · v0.23.2 · Apache-2.0
Fastest implementation of the most used tokenizers in AI.
Adoption
Downloads (90 days) 17.6M Observed downloads / 90 days
Popularity Top 25% on crates.io Derived 89th percentile within crates.io
Dependents 1K Observed packages that depend on it
All-time downloads 41.6M Observed since first publish
Maintenance 0.75 Derived Release cadence and recency, 0 to 1
Releases per year 6.1 43 versions over its life Growth by period Derived by Crest of Tech from two observations of crates.io’s own figure, a period apart. Nothing is stored or estimated: without an observation near the period’s start, the period is still collecting.
7 days Collecting history Collecting
30 days Collecting history Collecting
90 days Collecting history Collecting
1 year Collecting history Collecting Adoption history Observations in this range began OCT 5, 2026. The chart appears after 7 observations; 2 so far. Collecting history.
Table of observations (13) Day downloads / 90 days Dependents OCT 24, 2026 17,603,870 1,007 OCT 5, 2026 15,157,974 Not observed OCT 4, 2026 Not observed Not observed OCT 3, 2026 Not observed Not observed OCT 2, 2026 Not observed Not observed OCT 1, 2026 Not observed Not observed SEP 30, 2026 Not observed Not observed SEP 29, 2026 Not observed Not observed SEP 28, 2026 Not observed Not observed SEP 27, 2026 Not observed Not observed SEP 26, 2026 Not observed Not observed SEP 25, 2026 Not observed Not observed SEP 24, 2026 Not observed Not observed
Facts from the registry
Latest version v0.23.2 43 versions
First published AUG 8, 2019
Last release SEP 3, 2026 51d ago
Licence Apache-2.0 Repository In the Crest catalog
💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
Language Rust
Last push 24d ago
Last release 51d ago Dependencies latest version · v0.23.2 Runtime 24 ahash ^0.8.11compact_str ^0.9daachorse ^3.0.0dary_heap ^0.3.6derive_builder ^0.20esaxx-rs ^0.1.10getrandom ^0.3itertools ^0.14log ^0.4macro_rules_attribute ^0.2.0monostate ^0.1.12paste ^1.0.14rand ^0.9rayon ^1.10rayon-cond ^0.4regex ^1.10regex-syntax ^0.8serde ^1.0serde_json ^1.0spm_precompiled ^0.1.3thiserror ^2unicode-normalization-alignments ^0.1unicode-segmentation ^1.11unicode_categories ^0.1Optional 4 fancy-regex ^0.17hf-hub ^0.4.1indicatif ^0.18onig ^6.5.1bitcannon SIMD Unicode atom classification + bitstream pre-tokenization: one classify pass into byte-exact token spans (cl100k, o200k, GPT-2/ByteLevel, DeepSeek, Whitespace, Bert, ...). Published from the same repository tk_convert The legacy `tokenizer.json` upgrade pass of the 🤗 Tokenizers library: a JSON→JSON
rewrite that turns a config written by any older version into the canonical form
`tk-serialize`'s reader accepts. Published from the same repository tk_encode Inference (encode/decode) half of the 🤗 Tokenizers library: model engines,
the full pipeline components, and the `Tokenizer` orchestration — without any
training code. Published from the same repository tk_serialize The reader for the canonical `tokenizer.json` format: a `hifijson` parse and a
builder that constructs `tk-encode`'s runtime pipeline directly, with no serde
anywhere. Legacy config shapes are `tk-convert`'s job, not this crate's. Published from the same repository tokenizers Published from the same repository tokenizers Provides an implementation of today's most used tokenizers, with a focus on performances and versatility. Published from the same repository keras-hub-nightly Pretrained models for Keras. Same primary category nemo-relay Python bindings for the NeMo Relay agent runtime. Same primary category pipecat-ai An open source framework for voice (and multimodal) assistants Same primary category langflow-base A Python package with a built-in web application Same primary category compressed-tensors Library for utilization of compressed safetensors of neural network models Same primary category langgraph Building stateful, multi-actor applications with LLMs Same primary category 