In-loop evaluation tasks for language modeling
Project Links
Meta
Author: Allen Institute for Artificial Intelligence
Requires Python: >=3.9
Classifiers
OLMo-in-loop-evals
Code for in-loop evaluation tasks used by the OLMo training team.
Installation
pip install ai2-olmo-eval
Release process
Steps
-
Update the version in
src/olmo_eval/version.py. -
Run the release script:
./src/scripts/release.sh
This will commit the changes to the CHANGELOG and
version.pyfiles and then create a new tag in git which will trigger a workflow on GitHub Actions that handles the rest.
Fixing a failed release
If for some reason the GitHub Actions release workflow failed with an error that needs to be fixed, you'll have to delete the tag on GitHub. Once you've pushed a fix you can simply repeat the steps above.
0.9.0
Feb 04, 2026
0.8.7
Jan 28, 2026
0.8.6
Aug 12, 2025
0.8.5
Jul 20, 2025
0.8.4
Jun 05, 2025
0.8.3
May 27, 2025
0.8.2
May 19, 2025
0.8.1
May 18, 2025
0.8.0
May 18, 2025
0.7.2
May 16, 2025
0.7.1
Apr 03, 2025
0.7.0
Mar 10, 2025
0.6.1
Feb 10, 2025
0.6.0
Dec 19, 2024
0.5.0
Dec 19, 2024
0.3.0
Dec 18, 2024
0.2.0
Oct 29, 2024
0.1.0
Oct 28, 2024
