Packages crates.io tk_serialize Package · crates.io
tk_serialize crates.io · v0.1.0-rc.2 · Apache-2.0
The reader for the canonical `tokenizer.json` format: a `hifijson` parse and a
builder that constructs `tk-encode`'s runtime pipeline directly, with no serde
anywhere. Legacy config shapes are `tk-convert`'s job, not this crate's.
Install cargo add tk_serializeCopy Adoption
Downloads (90 days) 1.2K Observed downloads / 90 days
Popularity Lower half on crates.io Derived 1st percentile within crates.io
Dependents Collecting history Collecting
All-time downloads 1.2K Observed since first publish
Maintenance 0.95 Derived Release cadence and recency, 0 to 1
Releases per year 1095 3 versions over its life Growth by period Derived by Crest of Tech from two observations of crates.io’s own figure, a period apart. Nothing is stored or estimated: without an observation near the period’s start, the period is still collecting.
7 days Collecting history Collecting
30 days Collecting history Collecting
90 days Collecting history Collecting
1 year Collecting history Collecting Adoption history Observations in this range began OCT 6, 2026. The chart appears after 7 observations; 1 so far. Collecting history.
Table of observations (16) Day downloads / 90 days Dependents OCT 6, 2026 1,223 Not observed OCT 5, 2026 Not observed Not observed OCT 4, 2026 Not observed Not observed OCT 3, 2026 Not observed Not observed OCT 2, 2026 Not observed Not observed OCT 1, 2026 Not observed Not observed SEP 30, 2026 Not observed Not observed SEP 29, 2026 Not observed Not observed SEP 28, 2026 Not observed Not observed SEP 27, 2026 Not observed Not observed SEP 26, 2026 Not observed Not observed SEP 25, 2026 Not observed Not observed SEP 24, 2026 Not observed Not observed SEP 23, 2026 Not observed Not observed SEP 22, 2026 Not observed Not observed SEP 21, 2026 Not observed Not observed
Facts from the registry
Latest version v0.1.0-rc.2 3 versions
First published SEP 21, 2026
Last release SEP 21, 2026 33d ago
Licence Apache-2.0 Repository In the Crest catalog
💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
Language Rust
Last push 24d ago
Last release 51d ago Dependencies latest version · v0.1.0-rc.2 Runtime 4 base64 ^0.22bitcannon ^0.1.0-rc.2hifijson ^0.5.0tk-encode ^0.1.0-rc.2bitcannon SIMD Unicode atom classification + bitstream pre-tokenization: one classify pass into byte-exact token spans (cl100k, o200k, GPT-2/ByteLevel, DeepSeek, Whitespace, Bert, ...). Published from the same repository tk_convert The legacy `tokenizer.json` upgrade pass of the 🤗 Tokenizers library: a JSON→JSON
rewrite that turns a config written by any older version into the canonical form
`tk-serialize`'s reader accepts. Published from the same repository tk_encode Inference (encode/decode) half of the 🤗 Tokenizers library: model engines,
the full pipeline components, and the `Tokenizer` orchestration — without any
training code. Published from the same repository tokenizers Published from the same repository tokenizers Fastest implementation of the most used tokenizers in AI. Published from the same repository tokenizers Provides an implementation of today's most used tokenizers, with a focus on performances and versatility. Published from the same repository org.k2fsa.sherpa.onnx.runtime.osx-arm64 .NET native osx-arm64 wrapper for the sherpa-onnx project.
In general, you don't need to use this package directly.
Please use https://www.nuget.org/packages/org.k2fsa.sherpa.onnx instead Same primary category org.k2fsa.sherpa.onnx.runtime.linux-x64 .NET native linux-x64 wrapper for the sherpa-onnx project.
In general, you don't need to use this package directly.
Please use https://www.nuget.org/packages/org.k2fsa.sherpa.onnx instead Same primary category org.k2fsa.sherpa.onnx.runtime.linux-arm64 .NET native linux-arm64 wrapper for the sherpa-onnx project.
In general, you don't need to use this package directly.
Please use https://www.nuget.org/packages/org.k2fsa.sherpa.onnx instead Same primary category org.k2fsa.sherpa.onnx.runtime.android-x64 .NET native android-x64 wrapper for the sherpa-onnx project.
In general, you don't need to use this package directly.
Please use https://www.nuget.org/packages/org.k2fsa.sherpa.onnx instead Same primary category org.k2fsa.sherpa.onnx.runtime.android-arm64 .NET native android-arm64 wrapper for the sherpa-onnx project.
In general, you don't need to use this package directly.
Please use https://www.nuget.org/packages/org.k2fsa.sherpa.onnx instead Same primary category org.k2fsa.sherpa.onnx sherpa-onnx is an open-source real-time speech recognition toolkit developed
by the Next-gen Kaldi team. It supports streaming recognition on a variety of
platforms such as Android, iOS, Raspberry, Linux, Windows, macOS, etc.
It does not require Internet connection during recognition.
See the documentation https://k2-fsa.github.io/sherpa/onnx/index.html
for details. Same primary category 