A spaCy package for the Rust tokenizations library
Project Links
Meta
Author: Explosion
Requires Python: <3.14,>=3.9
Classifiers
Development Status
- 4 - Beta
Environment
- Console
Intended Audience
- Developers
- Science/Research
Topic
- Scientific/Engineering
- Scientific/Engineering :: Artificial Intelligence
License
- OSI Approved :: MIT License
Operating System
- POSIX :: Linux
- MacOS :: MacOS X
- Microsoft :: Windows
Programming Language
- Rust
- Python
- Python :: 3
- Python :: 3.9
- Python :: 3.10
- Python :: 3.11
- Python :: 3.12
- Python :: 3.12
spacy-alignments: Align tokenizations for spaCy + transformers
A spaCy package for Yohei Tamura's Rust tokenizations library with Python bindings.
Installation
pip install -U pip setuptools wheel
pip install spacy-alignments
If no binary wheel is available for your platform, you will need to install
Rust in order to build
spacy-alignments from source.
spacy-alignments vs. pytokenizations
The spacy_alignments module is a drop-in replacement for tokenizations:
import spacy_alignments as tokenizations
a2b, b2a = tokenizations.get_alignments(["å", "BC"], ["abc"])
assert a2b == [[0], [0]]
assert b2a == [[0, 1]]
The only difference between this package and the original
pytokenizations is that it
switches the build system to setuptools-rust to make it easier for us at
Explosion to build source and binary packages for a wider range of platforms.
Bug reports and other issues
Please use spaCy's issue tracker to report a bug, or open a new thread on the discussion board for any other issue.
0.9.2
Jun 03, 2025
0.9.1
Sep 25, 2023
0.9.0
Dec 19, 2022
0.8.6
Oct 17, 2022
0.8.5
Apr 06, 2022
0.8.4
Nov 08, 2021
0.8.3
Apr 09, 2021
0.7.2
Dec 08, 2020
Wheel compatibility matrix
Files in release
spacy_alignments-0.9.2-cp310-cp310-macosx_10_12_x86_64.whl (286.3KiB)
spacy_alignments-0.9.2-cp310-cp310-macosx_11_0_arm64.whl (282.3KiB)
spacy_alignments-0.9.2-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (307.2KiB)
spacy_alignments-0.9.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (307.1KiB)
spacy_alignments-0.9.2-cp310-cp310-musllinux_1_2_aarch64.whl (354.1KiB)
spacy_alignments-0.9.2-cp310-cp310-musllinux_1_2_x86_64.whl (367.2KiB)
spacy_alignments-0.9.2-cp310-cp310-win_amd64.whl (180.4KiB)
spacy_alignments-0.9.2-cp311-cp311-macosx_10_12_x86_64.whl (286.1KiB)
spacy_alignments-0.9.2-cp311-cp311-macosx_11_0_arm64.whl (282.1KiB)
spacy_alignments-0.9.2-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (307.2KiB)
spacy_alignments-0.9.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (306.8KiB)
spacy_alignments-0.9.2-cp311-cp311-musllinux_1_2_aarch64.whl (354.0KiB)
spacy_alignments-0.9.2-cp311-cp311-musllinux_1_2_x86_64.whl (366.9KiB)
spacy_alignments-0.9.2-cp311-cp311-win_amd64.whl (180.2KiB)
spacy_alignments-0.9.2-cp312-cp312-macosx_10_13_x86_64.whl (284.0KiB)
spacy_alignments-0.9.2-cp312-cp312-macosx_11_0_arm64.whl (279.5KiB)
spacy_alignments-0.9.2-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (306.7KiB)
spacy_alignments-0.9.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (306.1KiB)
spacy_alignments-0.9.2-cp312-cp312-musllinux_1_2_aarch64.whl (353.7KiB)
spacy_alignments-0.9.2-cp312-cp312-musllinux_1_2_x86_64.whl (365.9KiB)
spacy_alignments-0.9.2-cp312-cp312-win_amd64.whl (180.0KiB)
spacy_alignments-0.9.2-cp313-cp313-macosx_10_13_x86_64.whl (283.7KiB)
spacy_alignments-0.9.2-cp313-cp313-macosx_11_0_arm64.whl (279.2KiB)
spacy_alignments-0.9.2-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (306.7KiB)
spacy_alignments-0.9.2-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (306.1KiB)
spacy_alignments-0.9.2-cp313-cp313-musllinux_1_2_aarch64.whl (353.6KiB)
spacy_alignments-0.9.2-cp313-cp313-musllinux_1_2_x86_64.whl (365.9KiB)
spacy_alignments-0.9.2-cp313-cp313-win_amd64.whl (180.1KiB)
spacy_alignments-0.9.2-cp39-cp39-macosx_10_12_x86_64.whl (286.1KiB)
spacy_alignments-0.9.2-cp39-cp39-macosx_11_0_arm64.whl (282.2KiB)
spacy_alignments-0.9.2-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (307.5KiB)
spacy_alignments-0.9.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (307.3KiB)
spacy_alignments-0.9.2-cp39-cp39-musllinux_1_2_aarch64.whl (354.4KiB)
spacy_alignments-0.9.2-cp39-cp39-musllinux_1_2_x86_64.whl (367.4KiB)
spacy_alignments-0.9.2-cp39-cp39-win_amd64.whl (180.5KiB)
spacy_alignments-0.9.2.tar.gz (6.9KiB)
No dependencies