Packages npm tokenizers Package · npm
tokenizers npm · v0.23.2 · Apache-2.0
Provides an implementation of today's most used tokenizers, with a focus on performances and versatility.
Install npm install tokenizersCopy Adoption
Downloads (30 days) 229.8K Observed downloads / 30 days
Popularity Lower half on npm Derived 29th percentile within npm
Dependents Collecting history Collecting
Maintenance 0.75 Derived Release cadence and recency, 0 to 1
Releases per year 5.7 42 versions over its life Growth by period Derived by Crest of Tech from two observations of npm’s own figure, a period apart. Nothing is stored or estimated: without an observation near the period’s start, the period is still collecting.
7 days +5.2% · 7d Derived +11.3K downloads / 30 days, SEP 30 → OCT 7
30 days +171.7% · 30d Derived +145.2K downloads / 30 days, SEP 7 → OCT 7
90 days +150.6% · 90d Derived +138.1K downloads / 30 days, JUL 6 → OCT 7
1 year +2959.2% · 1y Derived +222.3K downloads / 30 days, NOV 10 → OCT 7 Adoption history Adoption historydownloads / 30 days · 31 observations
30d 90d 1y 2y 238.4K 83.8K SEP 7, 2026: 84,559 downloads / 30 days SEP 8, 2026: 83,768 downloads / 30 days SEP 9, 2026: 86,063 downloads / 30 days SEP 10, 2026: 88,958 downloads / 30 days SEP 11, 2026: 92,732 downloads / 30 days SEP 12, 2026: 92,349 downloads / 30 days SEP 13, 2026: 94,899 downloads / 30 days SEP 14, 2026: 99,810 downloads / 30 days SEP 15, 2026: 97,167 downloads / 30 days SEP 16, 2026: 103,000 downloads / 30 days SEP 17, 2026: 111,737 downloads / 30 days SEP 18, 2026: 119,977 downloads / 30 days SEP 19, 2026: 119,555 downloads / 30 days SEP 20, 2026: 121,355 downloads / 30 days SEP 21, 2026: 132,121 downloads / 30 days SEP 22, 2026: 142,953 downloads / 30 days SEP 23, 2026: 153,222 downloads / 30 days SEP 24, 2026: 162,516 downloads / 30 days SEP 25, 2026: 174,720 downloads / 30 days SEP 26, 2026: 180,276 downloads / 30 days SEP 27, 2026: 183,393 downloads / 30 days SEP 28, 2026: 197,433 downloads / 30 days SEP 29, 2026: 211,468 downloads / 30 days SEP 30, 2026: 218,489 downloads / 30 days OCT 1, 2026: 225,050 downloads / 30 days OCT 2, 2026: 229,025 downloads / 30 days OCT 3, 2026: 233,425 downloads / 30 days OCT 4, 2026: 229,774 downloads / 30 days OCT 5, 2026: 238,433 downloads / 30 days OCT 6, 2026: 235,717 downloads / 30 days OCT 7, 2026: 229,774 downloads / 30 days SEP 7 SEP 22 OCT 7
Table of observations (31) Day downloads / 30 days Dependents OCT 7, 2026 229,774 Not observed OCT 6, 2026 235,717 Not observed OCT 5, 2026 238,433 Not observed OCT 4, 2026 229,774 Not observed OCT 3, 2026 233,425 Not observed OCT 2, 2026 229,025 Not observed OCT 1, 2026 225,050 Not observed SEP 30, 2026 218,489 Not observed SEP 29, 2026 211,468 Not observed SEP 28, 2026 197,433 Not observed SEP 27, 2026 183,393 Not observed SEP 26, 2026 180,276 Not observed SEP 25, 2026 174,720 Not observed SEP 24, 2026 162,516 Not observed SEP 23, 2026 153,222 Not observed SEP 22, 2026 142,953 Not observed SEP 21, 2026 132,121 Not observed SEP 20, 2026 121,355 Not observed SEP 19, 2026 119,555 Not observed SEP 18, 2026 119,977 Not observed SEP 17, 2026 111,737 Not observed SEP 16, 2026 103,000 Not observed SEP 15, 2026 97,167 Not observed SEP 14, 2026 99,810 Not observed SEP 13, 2026 94,899 Not observed SEP 12, 2026 92,349 Not observed SEP 11, 2026 92,732 Not observed SEP 10, 2026 88,958 Not observed SEP 9, 2026 86,063 Not observed SEP 8, 2026 83,768 Not observed SEP 7, 2026 84,559 Not observed
Facts from the registry
Latest version v0.23.2 42 versions
First published MAY 15, 2019
Last release SEP 3, 2026 51d ago
Licence Apache-2.0 Repository In the Crest catalog
💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
Language Rust
Last push 24d ago
Last release 51d ago Dependencies latest version · v0.23.2 No dependencies are recorded for the latest version.
bitcannon SIMD Unicode atom classification + bitstream pre-tokenization: one classify pass into byte-exact token spans (cl100k, o200k, GPT-2/ByteLevel, DeepSeek, Whitespace, Bert, ...). Published from the same repository tk_convert The legacy `tokenizer.json` upgrade pass of the 🤗 Tokenizers library: a JSON→JSON
rewrite that turns a config written by any older version into the canonical form
`tk-serialize`'s reader accepts. Published from the same repository tk_encode Inference (encode/decode) half of the 🤗 Tokenizers library: model engines,
the full pipeline components, and the `Tokenizer` orchestration — without any
training code. Published from the same repository tk_serialize The reader for the canonical `tokenizer.json` format: a `hifijson` parse and a
builder that constructs `tk-encode`'s runtime pipeline directly, with no serde
anywhere. Legacy config shapes are `tk-convert`'s job, not this crate's. Published from the same repository tokenizers Fastest implementation of the most used tokenizers in AI. Published from the same repository tokenizers Published from the same repository org.k2fsa.sherpa.onnx.runtime.osx-arm64 .NET native osx-arm64 wrapper for the sherpa-onnx project.
In general, you don't need to use this package directly.
Please use https://www.nuget.org/packages/org.k2fsa.sherpa.onnx instead Same primary category org.k2fsa.sherpa.onnx.runtime.linux-x64 .NET native linux-x64 wrapper for the sherpa-onnx project.
In general, you don't need to use this package directly.
Please use https://www.nuget.org/packages/org.k2fsa.sherpa.onnx instead Same primary category org.k2fsa.sherpa.onnx.runtime.linux-arm64 .NET native linux-arm64 wrapper for the sherpa-onnx project.
In general, you don't need to use this package directly.
Please use https://www.nuget.org/packages/org.k2fsa.sherpa.onnx instead Same primary category org.k2fsa.sherpa.onnx.runtime.android-x64 .NET native android-x64 wrapper for the sherpa-onnx project.
In general, you don't need to use this package directly.
Please use https://www.nuget.org/packages/org.k2fsa.sherpa.onnx instead Same primary category org.k2fsa.sherpa.onnx.runtime.android-arm64 .NET native android-arm64 wrapper for the sherpa-onnx project.
In general, you don't need to use this package directly.
Please use https://www.nuget.org/packages/org.k2fsa.sherpa.onnx instead Same primary category org.k2fsa.sherpa.onnx sherpa-onnx is an open-source real-time speech recognition toolkit developed
by the Next-gen Kaldi team. It supports streaming recognition on a variety of
platforms such as Android, iOS, Raspberry, Linux, Windows, macOS, etc.
It does not require Internet connection during recognition.
See the documentation https://k2-fsa.github.io/sherpa/onnx/index.html
for details. Same primary category 