Mistral-common

What is it?

mistral-common is a set of tools to help you work with Mistral AI models.

We open-source the tokenizers, validation and normalization code that can be used with our models.

This ensures that you can take full advantage of our models for the following features:

• tokenization of text, images and tools calls.
• validation and normalization of requests, messages, tool calls, and responses. This is built on top of the Pydantic library.

We also version our tokenizers to guarantee backward compatibility for the models that we release.

For who ?

This library is for you if you want to:

• use our models in your own application.
• build your own models and want to use the same tokenization and validation code as we do.

How to use it ?

You can install the library using pip:

pip install mistral-common

We propose different dependencies to install depending on your needs:

• image: to use the image tokenizers.
• audio: to use the audio tokenizers.
• hf-hub: to download the tokenizers from the Hugging Face Hub.
• sentencepiece: to allow the use of SentencePiece tokenizers. This is now optional as we only release Tekken tokenizers for recent models.
• [Experimental] server: to use our tokenizers in a server mode.

Each dependency is optional and can be installed separately or all together using the following commands:

pip install "mistral-common[image]"
pip install "mistral-common[audio]"
pip install "mistral-common[hf-hub]"
pip install "mistral-common[sentencepiece]"
pip install "mistral-common[server]"
pip install "mistral-common[image,audio,hf-hub,sentencepiece,server]"

For more information, please refer to the documentation.

How to contribute?

We welcome contributions to this library. Please read our contribution guide first. If you encounter a bug, have difficulties using mistral-common, or want to propose a feature, search the existing issues and open an issue on our GitHub repository before starting work.

If you want to contribute code after the issue discussion, you may fork the repository and open a pull request. We will review changes that are relevant to the library and meet the contribution guidelines.

# Clone the repository
git clone https://github.com/<your_fork_username>/mistral-common.git
cd mistral-common

# Create a virtual environment and install the dependencies
uv venv
source .venv/bin/activate
uv sync --frozen --all-extras --group dev # --group docs if you want to build the documentation

# Install the pre-commit hooks
uv run pre-commit install

License

This library is licensed under the Apache 2.0 License. See the LICENSE file for more information.

You must not use this library or our models in a manner that infringes, misappropriates, or otherwise violates any third party’s rights, including intellectual property rights.