tokenizers

Install

cargo add tokenizers

Adoption

Downloads (90 days)

17.6M

Observeddownloads / 90 days

Popularity

Top 25% on crates.io

Derived89th percentile within crates.io

Dependents

1K

Observedpackages that depend on it

All-time downloads

41.6M

Observedsince first publish

Maintenance

0.75

DerivedRelease cadence and recency, 0 to 1

Releases per year

6.1

43 versions over its life

Growth by period

Derived by Crest of Tech from two observations of crates.io’s own figure, a period apart. Nothing is stored or estimated: without an observation near the period’s start, the period is still collecting.

7 days

Collectinghistory

Collecting

30 days

Collectinghistory

Collecting

90 days

Collectinghistory

Collecting

1 year

Collectinghistory

Collecting

Adoption history

Facts from the registry

Latest version

v0.23.2

43 versions

First published

AUG 8, 2019

Last release

SEP 3, 2026

51d ago

Licence

Apache-2.0

Links

crates.io page (opens in a new tab)

Homepage · github.com (opens in a new tab)

Docs · docs.rs (opens in a new tab)

Repository

Dependencies latest version · v0.23.2

Related packages

• bitcannonSIMD Unicode atom classification + bitstream pre-tokenization: one classify pass into byte-exact token spans (cl100k, o200k, GPT-2/ByteLevel, DeepSeek, Whitespace, Bert, ...). crates.ioPublished from the same repository
• tk_convertThe legacy `tokenizer.json` upgrade pass of the 🤗 Tokenizers library: a JSON→JSON rewrite that turns a config written by any older version into the canonical form `tk-serialize`'s reader accepts. crates.ioPublished from the same repository
• tk_encodeInference (encode/decode) half of the 🤗 Tokenizers library: model engines, the full pipeline components, and the `Tokenizer` orchestration — without any training code. crates.ioPublished from the same repository
• tk_serializeThe reader for the canonical `tokenizer.json` format: a `hifijson` parse and a builder that constructs `tk-encode`'s runtime pipeline directly, with no serde anywhere. Legacy config shapes are `tk-convert`'s job, not this crate's. crates.ioPublished from the same repository
• tokenizers PyPIPublished from the same repository
• tokenizersProvides an implementation of today's most used tokenizers, with a focus on performances and versatility. npmPublished from the same repository
• keras-hub-nightlyPretrained models for Keras. PyPISame primary category
• nemo-relayPython bindings for the NeMo Relay agent runtime. PyPISame primary category
• pipecat-aiAn open source framework for voice (and multimodal) assistants PyPISame primary category
• langflow-baseA Python package with a built-in web application PyPISame primary category
• compressed-tensorsLibrary for utilization of compressed safetensors of neural network models PyPISame primary category
• langgraphBuilding stateful, multi-actor applications with LLMs PyPISame primary category

Related news

• Launchtokenizers v1: encode, decode and scaling, measured
  33d agoSEP 21, 2026