Warning: This is a tech preview of github.com/simple-repository. The service is unsupported and may be removed at any time.

Project Links

Meta

Author: Explosion

Requires Python: >=3.9

Classifiers

Development Status

5 - Production/Stable

Environment

Console

Intended Audience

Developers
Science/Research

License

OSI Approved :: MIT License

Operating System

MacOS :: MacOS X

Programming Language

Cython
Python :: 3
Python :: 3.9
Python :: 3.10
Python :: 3.11
Python :: 3.12
Python :: 3.13
Python :: 3.14

Topic

Scientific/Engineering

🥢 Curated Tokenizers

This Python library provides word-/sentencepiece tokenizers. The following types of tokenizers are currenty supported:

Tokenizer	Binding	Example model
BPE	sentencepiece
Byte BPE	Native	RoBERTa/GPT-2
Unigram	sentencepiece	XLM-RoBERTa
Wordpiece	Native	BERT

⚠️ Warning: experimental package

This package is experimental and it is likely that the APIs will change in incompatible ways.

⏳ Install

Curated tokenizers is availble through PyPI:

pip install curated_tokenizers

🚀 Quickstart

The best way to get started with curated tokenizers is through the curated-transformers library. curated-transformers also provides functionality to load tokenization models from Huggingface Hub.

2.0.1 Aug 24, 2026

2.0.0 Apr 15, 2024

2.0.0.dev0 Apr 12, 2024

0.9.2 Feb 07, 2024

0.9.1 Sep 14, 2023

0.9.0 Jul 11, 2023

0.9.0.dev0 Jul 06, 2023

0.0.10 Aug 24, 2026

0.0.9 Jan 18, 2024

0.0.8 Jul 06, 2023

0.0.7 May 23, 2023

0.0.6 May 23, 2023

Wheel compatibility matrix

Platform	CPython 3.9	CPython 3.10	CPython 3.11	CPython 3.12	CPython 3.13	CPython 3.14	CPython (additional flags: t) 3.14
macosx_10_13_x86_64
macosx_10_15_x86_64
macosx_10_9_x86_64
macosx_11_0_arm64
manylinux_2_27_aarch64
manylinux_2_27_x86_64
manylinux_2_28_aarch64
manylinux_2_28_x86_64
musllinux_1_2_aarch64
musllinux_1_2_x86_64
win32
win_amd64
win_arm64

Files in release

curated_tokenizers-2.0.1-cp310-cp310-macosx_10_9_x86_64.whl (721.5KiB)

curated_tokenizers-2.0.1-cp310-cp310-macosx_11_0_arm64.whl (700.3KiB)

curated_tokenizers-2.0.1-cp310-cp310-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl (6.0MiB)

curated_tokenizers-2.0.1-cp310-cp310-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (6.1MiB)

curated_tokenizers-2.0.1-cp310-cp310-musllinux_1_2_aarch64.whl (6.7MiB)

curated_tokenizers-2.0.1-cp310-cp310-musllinux_1_2_x86_64.whl (6.9MiB)

curated_tokenizers-2.0.1-cp310-cp310-win32.whl (712.2KiB)

curated_tokenizers-2.0.1-cp310-cp310-win_amd64.whl (759.3KiB)

curated_tokenizers-2.0.1-cp310-cp310-win_arm64.whl (729.2KiB)

curated_tokenizers-2.0.1-cp311-cp311-macosx_10_9_x86_64.whl (723.5KiB)

curated_tokenizers-2.0.1-cp311-cp311-macosx_11_0_arm64.whl (702.7KiB)

curated_tokenizers-2.0.1-cp311-cp311-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl (6.1MiB)

curated_tokenizers-2.0.1-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (6.2MiB)

curated_tokenizers-2.0.1-cp311-cp311-musllinux_1_2_aarch64.whl (6.8MiB)

curated_tokenizers-2.0.1-cp311-cp311-musllinux_1_2_x86_64.whl (6.9MiB)

curated_tokenizers-2.0.1-cp311-cp311-win32.whl (711.4KiB)

curated_tokenizers-2.0.1-cp311-cp311-win_amd64.whl (759.5KiB)

curated_tokenizers-2.0.1-cp311-cp311-win_arm64.whl (728.7KiB)

curated_tokenizers-2.0.1-cp312-cp312-macosx_10_13_x86_64.whl (725.2KiB)

curated_tokenizers-2.0.1-cp312-cp312-macosx_11_0_arm64.whl (704.1KiB)

curated_tokenizers-2.0.1-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl (6.1MiB)

curated_tokenizers-2.0.1-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (6.2MiB)

curated_tokenizers-2.0.1-cp312-cp312-musllinux_1_2_aarch64.whl (6.8MiB)

curated_tokenizers-2.0.1-cp312-cp312-musllinux_1_2_x86_64.whl (7.0MiB)

curated_tokenizers-2.0.1-cp312-cp312-win32.whl (712.1KiB)

curated_tokenizers-2.0.1-cp312-cp312-win_amd64.whl (760.4KiB)

curated_tokenizers-2.0.1-cp312-cp312-win_arm64.whl (727.7KiB)

curated_tokenizers-2.0.1-cp313-cp313-macosx_10_13_x86_64.whl (723.1KiB)

curated_tokenizers-2.0.1-cp313-cp313-macosx_11_0_arm64.whl (702.1KiB)

curated_tokenizers-2.0.1-cp313-cp313-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl (6.1MiB)

curated_tokenizers-2.0.1-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (6.2MiB)

curated_tokenizers-2.0.1-cp313-cp313-musllinux_1_2_aarch64.whl (6.8MiB)

curated_tokenizers-2.0.1-cp313-cp313-musllinux_1_2_x86_64.whl (7.0MiB)

curated_tokenizers-2.0.1-cp313-cp313-win32.whl (711.7KiB)

curated_tokenizers-2.0.1-cp313-cp313-win_amd64.whl (759.7KiB)

curated_tokenizers-2.0.1-cp313-cp313-win_arm64.whl (727.6KiB)

curated_tokenizers-2.0.1-cp314-cp314-macosx_10_15_x86_64.whl (723.0KiB)

curated_tokenizers-2.0.1-cp314-cp314-macosx_11_0_arm64.whl (702.1KiB)

curated_tokenizers-2.0.1-cp314-cp314-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl (6.1MiB)

curated_tokenizers-2.0.1-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (6.2MiB)

curated_tokenizers-2.0.1-cp314-cp314-musllinux_1_2_aarch64.whl (6.8MiB)

curated_tokenizers-2.0.1-cp314-cp314-musllinux_1_2_x86_64.whl (7.0MiB)

curated_tokenizers-2.0.1-cp314-cp314-win32.whl (738.1KiB)

curated_tokenizers-2.0.1-cp314-cp314-win_amd64.whl (787.4KiB)

curated_tokenizers-2.0.1-cp314-cp314-win_arm64.whl (755.7KiB)

curated_tokenizers-2.0.1-cp314-cp314t-macosx_10_15_x86_64.whl (730.0KiB)

curated_tokenizers-2.0.1-cp314-cp314t-macosx_11_0_arm64.whl (711.3KiB)

curated_tokenizers-2.0.1-cp314-cp314t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl (6.2MiB)

curated_tokenizers-2.0.1-cp314-cp314t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (6.3MiB)

curated_tokenizers-2.0.1-cp314-cp314t-musllinux_1_2_aarch64.whl (6.9MiB)

curated_tokenizers-2.0.1-cp314-cp314t-musllinux_1_2_x86_64.whl (7.0MiB)

curated_tokenizers-2.0.1-cp314-cp314t-win32.whl (748.4KiB)

curated_tokenizers-2.0.1-cp314-cp314t-win_amd64.whl (796.9KiB)

curated_tokenizers-2.0.1-cp314-cp314t-win_arm64.whl (763.8KiB)

curated_tokenizers-2.0.1-cp39-cp39-macosx_10_9_x86_64.whl (723.0KiB)

curated_tokenizers-2.0.1-cp39-cp39-macosx_11_0_arm64.whl (702.0KiB)

curated_tokenizers-2.0.1-cp39-cp39-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl (6.0MiB)

curated_tokenizers-2.0.1-cp39-cp39-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (6.1MiB)

curated_tokenizers-2.0.1-cp39-cp39-musllinux_1_2_aarch64.whl (6.7MiB)

curated_tokenizers-2.0.1-cp39-cp39-musllinux_1_2_x86_64.whl (6.9MiB)

curated_tokenizers-2.0.1-cp39-cp39-win32.whl (713.1KiB)

curated_tokenizers-2.0.1-cp39-cp39-win_amd64.whl (760.2KiB)

curated_tokenizers-2.0.1-cp39-cp39-win_arm64.whl (730.0KiB)

curated_tokenizers-2.0.1.tar.gz (2.2MiB)

Extras: None

Dependencies: