tokenizers

Install

npm install tokenizers

Adoption

Downloads (30 days)

229.8K

Observeddownloads / 30 days

Popularity

Lower half on npm

Derived29th percentile within npm

Dependents

Collectinghistory

Collecting

Maintenance

0.75

DerivedRelease cadence and recency, 0 to 1

Releases per year

5.7

42 versions over its life

Growth by period

Derived by Crest of Tech from two observations of npm’s own figure, a period apart. Nothing is stored or estimated: without an observation near the period’s start, the period is still collecting.

7 days

+5.2% · 7d

Derived+11.3K downloads / 30 days, SEP 30 → OCT 7

30 days

+171.7% · 30d

Derived+145.2K downloads / 30 days, SEP 7 → OCT 7

90 days

+150.6% · 90d

Derived+138.1K downloads / 30 days, JUL 6 → OCT 7

1 year

+2959.2% · 1y

Derived+222.3K downloads / 30 days, NOV 10 → OCT 7

Adoption history

Facts from the registry

Latest version

v0.23.2

42 versions

First published

MAY 15, 2019

Last release

SEP 3, 2026

51d ago

Licence

Apache-2.0

Links

npm page (opens in a new tab)

Homepage · github.com (opens in a new tab)

Source · github.com (opens in a new tab)

Repository

Dependencies latest version · v0.23.2

No dependencies are recorded for the latest version.

Related packages

• bitcannonSIMD Unicode atom classification + bitstream pre-tokenization: one classify pass into byte-exact token spans (cl100k, o200k, GPT-2/ByteLevel, DeepSeek, Whitespace, Bert, ...). crates.ioPublished from the same repository
• tk_convertThe legacy `tokenizer.json` upgrade pass of the 🤗 Tokenizers library: a JSON→JSON rewrite that turns a config written by any older version into the canonical form `tk-serialize`'s reader accepts. crates.ioPublished from the same repository
• tk_encodeInference (encode/decode) half of the 🤗 Tokenizers library: model engines, the full pipeline components, and the `Tokenizer` orchestration — without any training code. crates.ioPublished from the same repository
• tk_serializeThe reader for the canonical `tokenizer.json` format: a `hifijson` parse and a builder that constructs `tk-encode`'s runtime pipeline directly, with no serde anywhere. Legacy config shapes are `tk-convert`'s job, not this crate's. crates.ioPublished from the same repository
• tokenizersFastest implementation of the most used tokenizers in AI. crates.ioPublished from the same repository
• tokenizers PyPIPublished from the same repository
• org.k2fsa.sherpa.onnx.runtime.osx-arm64.NET native osx-arm64 wrapper for the sherpa-onnx project. In general, you don't need to use this package directly. Please use https://www.nuget.org/packages/org.k2fsa.sherpa.onnx instead NuGetSame primary category
• org.k2fsa.sherpa.onnx.runtime.linux-x64.NET native linux-x64 wrapper for the sherpa-onnx project. In general, you don't need to use this package directly. Please use https://www.nuget.org/packages/org.k2fsa.sherpa.onnx instead NuGetSame primary category
• org.k2fsa.sherpa.onnx.runtime.linux-arm64.NET native linux-arm64 wrapper for the sherpa-onnx project. In general, you don't need to use this package directly. Please use https://www.nuget.org/packages/org.k2fsa.sherpa.onnx instead NuGetSame primary category
• org.k2fsa.sherpa.onnx.runtime.android-x64.NET native android-x64 wrapper for the sherpa-onnx project. In general, you don't need to use this package directly. Please use https://www.nuget.org/packages/org.k2fsa.sherpa.onnx instead NuGetSame primary category
• org.k2fsa.sherpa.onnx.runtime.android-arm64.NET native android-arm64 wrapper for the sherpa-onnx project. In general, you don't need to use this package directly. Please use https://www.nuget.org/packages/org.k2fsa.sherpa.onnx instead NuGetSame primary category
• org.k2fsa.sherpa.onnxsherpa-onnx is an open-source real-time speech recognition toolkit developed by the Next-gen Kaldi team. It supports streaming recognition on a variety of platforms such as Android, iOS, Raspberry, Linux, Windows, macOS, etc. It does not require Internet connection during recognition. See the documentation https://k2-fsa.github.io/sherpa/onnx/index.html for details. NuGetSame primary category

Related news

• Launchtokenizers v1: encode, decode and scaling, measured
  33d agoSEP 21, 2026