Packages PyPI tokenizers Package · PyPI
tokenizers PyPI · v0.23.3 · Apache Software License
The registry carries no description for this package.
Install pip install tokenizersCopy Adoption
Adoption — Not published by PyPI Not published
Popularity — Not published by PyPI Not published
Dependents — Not published by PyPI Not published
Maintenance 0.92 Derived Release cadence and recency, 0 to 1
Releases per year 14 100 versions over its life Growth by period PyPI publishes no adoption figure, so there is no growth to derive. Nothing is estimated in its place.
7 days — Not published by PyPI Not published
30 days — Not published by PyPI Not published
90 days — Not published by PyPI Not published
1 year — Not published by PyPI Not published Adoption history PyPI publishes no adoption count, so there is no history to chart. That is a fact about the registry, not a gap in ours: the package’s releases and maintenance are measured instead.
Facts from the registry
Latest version v0.23.3 100 versions
First published NOV 1, 2019
Last release OCT 19, 2026 5d ago
Licence Apache Software License Repository In the Crest catalog
💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
Language Rust
Last push 24d ago
Last release 51d ago Dependencies latest version · v0.23.3 Runtime 1 huggingface-hub <3.0,>=0.16.4bitcannon SIMD Unicode atom classification + bitstream pre-tokenization: one classify pass into byte-exact token spans (cl100k, o200k, GPT-2/ByteLevel, DeepSeek, Whitespace, Bert, ...). Published from the same repository tk_convert The legacy `tokenizer.json` upgrade pass of the 🤗 Tokenizers library: a JSON→JSON
rewrite that turns a config written by any older version into the canonical form
`tk-serialize`'s reader accepts. Published from the same repository tk_encode Inference (encode/decode) half of the 🤗 Tokenizers library: model engines,
the full pipeline components, and the `Tokenizer` orchestration — without any
training code. Published from the same repository tk_serialize The reader for the canonical `tokenizer.json` format: a `hifijson` parse and a
builder that constructs `tk-encode`'s runtime pipeline directly, with no serde
anywhere. Legacy config shapes are `tk-convert`'s job, not this crate's. Published from the same repository tokenizers Fastest implementation of the most used tokenizers in AI. Published from the same repository tokenizers Provides an implementation of today's most used tokenizers, with a focus on performances and versatility. Published from the same repository ttnn General compute framework for Tenstorrent devices Same primary category keras-hub-nightly Pretrained models for Keras. Same primary category amd-gaia GAIA is a lightweight agent framework designed for the edge and AI PCs. Same primary category nemo-relay Python bindings for the NeMo Relay agent runtime. Same primary category pipecat-ai An open source framework for voice (and multimodal) assistants Same primary category langflow-base A Python package with a built-in web application Same primary category 