pytokens 0.4.1


pip install pytokens

  Latest version

Released: Jan 30, 2026
Navigation
Project Links
Meta
Author: Tushar Sadhwani
Requires Python: >=3.8
Classifiers

License
  • OSI Approved :: MIT License

Operating System
  • OS Independent

Programming Language
  • Python :: 3
  • Python :: 3 :: Only
  • Python :: 3.8
  • Python :: 3.9
  • Python :: 3.10
  • Python :: 3.11
  • Python :: 3.12
  • Python :: 3.13
  • Python :: 3.14
  • Python :: Implementation :: CPython

Typing
  • Typed

pytokens

A Fast, spec compliant Python 3.14+ tokenizer that runs on older Pythons.

Installation

pip install pytokens

Usage

python -m pytokens path/to/file.py

Local Development / Testing

  • Create and activate a virtual environment
  • Run PYTOKENS_USE_MYPYC=0 pip install -e '.[dev]' to do an editable install
  • Run pytest to run tests

Type Checking

Run mypy .

Compilation with mypyc

By default, we compile with mypyc. Use the PYTOKENS_USE_MYPYC environment variable to control this.

To check if you are using a compiled version, see whether the output of this is .py or .so:

python -c "import pytokens; print(pytokens.__file__)"

Create and upload a package to PyPI

  1. Make sure to bump the version in pyproject.toml.
  2. Push to Github so CI can build the wheels and sdist.
  3. Download the artifacts from the CI run.
  • Find the "Build wheels" job, click "Summary", scroll down to the bottom to see "Artifacts" and download the cibw-wheels artifact.
  1. Unzip the artifact.
  2. Upload the contents of the artifact to PyPI
  • e.g. via twine upload cibw-wheels/*
0.4.1 Jan 30, 2026
0.4.0 Jan 19, 2026
0.3.0 Nov 05, 2025
0.2.0 Oct 15, 2025
0.1.10 Feb 19, 2025
0.1.9 Jan 09, 2025
0.1.8 Jan 09, 2025
0.1.7 Jan 08, 2025
0.1.5 Dec 25, 2024
0.1.4 Dec 25, 2024
0.1.3.post1 Dec 23, 2024
0.1.3 Dec 23, 2024
0.1.2 Dec 20, 2024
0.1.1 Dec 20, 2024
0.1.0 Dec 20, 2024
0.0.0 Dec 13, 2024

Wheel compatibility matrix

Platform CPython 3.8 CPython 3.9 CPython 3.10 CPython 3.11 CPython 3.12 CPython 3.13 CPython 3.14 CPython (additional flags: t) 3.14 Python 3
any
macosx_11_0_arm64
manylinux2014_aarch64
manylinux2014_x86_64
manylinux_2_17_aarch64
manylinux_2_17_x86_64
manylinux_2_28_aarch64
manylinux_2_28_x86_64
musllinux_1_2_x86_64
win_amd64

Files in release

pytokens-0.4.1-cp310-cp310-macosx_11_0_arm64.whl (157.7KiB)
pytokens-0.4.1-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl (241.2KiB)
pytokens-0.4.1-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (253.4KiB)
pytokens-0.4.1-cp310-cp310-musllinux_1_2_x86_64.whl (256.5KiB)
pytokens-0.4.1-cp310-cp310-win_amd64.whl (101.1KiB)
pytokens-0.4.1-cp311-cp311-macosx_11_0_arm64.whl (157.1KiB)
pytokens-0.4.1-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl (242.7KiB)
pytokens-0.4.1-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (254.7KiB)
pytokens-0.4.1-cp311-cp311-musllinux_1_2_x86_64.whl (257.9KiB)
pytokens-0.4.1-cp311-cp311-win_amd64.whl (100.9KiB)
pytokens-0.4.1-cp312-cp312-macosx_11_0_arm64.whl (156.9KiB)
pytokens-0.4.1-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl (249.6KiB)
pytokens-0.4.1-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (263.5KiB)
pytokens-0.4.1-cp312-cp312-musllinux_1_2_x86_64.whl (261.8KiB)
pytokens-0.4.1-cp312-cp312-win_amd64.whl (101.1KiB)
pytokens-0.4.1-cp313-cp313-macosx_11_0_arm64.whl (157.0KiB)
pytokens-0.4.1-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl (248.2KiB)
pytokens-0.4.1-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (262.1KiB)
pytokens-0.4.1-cp313-cp313-musllinux_1_2_x86_64.whl (260.6KiB)
pytokens-0.4.1-cp313-cp313-win_amd64.whl (101.1KiB)
pytokens-0.4.1-cp314-cp314-macosx_11_0_arm64.whl (157.1KiB)
pytokens-0.4.1-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl (248.3KiB)
pytokens-0.4.1-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (261.8KiB)
pytokens-0.4.1-cp314-cp314-musllinux_1_2_x86_64.whl (265.3KiB)
pytokens-0.4.1-cp314-cp314-win_amd64.whl (102.1KiB)
pytokens-0.4.1-cp314-cp314t-macosx_11_0_arm64.whl (164.5KiB)
pytokens-0.4.1-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl (283.7KiB)
pytokens-0.4.1-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (296.4KiB)
pytokens-0.4.1-cp314-cp314t-musllinux_1_2_x86_64.whl (299.5KiB)
pytokens-0.4.1-cp314-cp314t-win_amd64.whl (113.6KiB)
pytokens-0.4.1-cp38-cp38-macosx_11_0_arm64.whl (151.2KiB)
pytokens-0.4.1-cp38-cp38-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl (233.0KiB)
pytokens-0.4.1-cp38-cp38-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (246.6KiB)
pytokens-0.4.1-cp38-cp38-musllinux_1_2_x86_64.whl (249.7KiB)
pytokens-0.4.1-cp38-cp38-win_amd64.whl (101.9KiB)
pytokens-0.4.1-cp39-cp39-macosx_11_0_arm64.whl (157.7KiB)
pytokens-0.4.1-cp39-cp39-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl (240.4KiB)
pytokens-0.4.1-cp39-cp39-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (253.0KiB)
pytokens-0.4.1-cp39-cp39-musllinux_1_2_x86_64.whl (256.1KiB)
pytokens-0.4.1-cp39-cp39-win_amd64.whl (101.1KiB)
pytokens-0.4.1-py3-none-any.whl (13.4KiB)
pytokens-0.4.1.tar.gz (22.5KiB)
Extras:
Dependencies: