A framework for evaluating and optimizing agents and models using sandboxed environments.
Project Links
Meta
Author: Alex Shaw
Requires Python: >=3.12
Classifiers
Harbor
Harbor is a framework from the creators of Terminal-Bench for evaluating and optimizing agents and language models. You can use Harbor to:
- Evaluate arbitrary agents like Claude Code, OpenHands, Codex CLI, and more.
- Build and share your own benchmarks and environments.
- Conduct experiments in thousands of environments in parallel through providers like Daytona, Modal, LangSmith, Blaxel, Novita Sandbox, Tensorlake, and Runta.
- Generate rollouts for RL optimization.
Check out the Harbor Cookbook for end-to-end examples and guides.
Installation
uv tool install harbor
or
pip install harbor
Example: Running Terminal-Bench-2.0
Harbor is the official harness for Terminal-Bench-2.0:
export ANTHROPIC_API_KEY=<YOUR-KEY>
harbor run --dataset terminal-bench@2.0 \
--agent claude-code \
--model anthropic/claude-opus-4-1 \
--n-concurrent 4
This will launch the benchmark locally using Docker. To run it on a cloud provider (like Daytona) pass the --env flag as below:
export ANTHROPIC_API_KEY=<YOUR-KEY>
export DAYTONA_API_KEY=<YOUR-KEY>
harbor run --dataset terminal-bench@2.0 \
--agent claude-code \
--model anthropic/claude-opus-4-1 \
--n-concurrent 100 \
--env daytona
To see all supported agents, and other options run:
harbor run --help
To explore all supported third party benchmarks (like SWE-Bench and Aider Polyglot) run:
harbor datasets list
To evaluate an agent and model one of these datasets, you can use the following command:
harbor run -d "<dataset@version>" -m "<model>" -a "<agent>"
Citation
If you use Harbor in academic work, please cite it using the “Cite this repository” button on GitHub or the following BibTeX entry:
@software{Harbor_Framework,
author = {{Harbor Framework Team}},
title = {{Harbor: A framework for evaluating and optimizing agents and models in container environments}},
year = {2026},
doi = {10.5281/zenodo.20953922},
url = {https://doi.org/10.5281/zenodo.20953922}
}
The DOI above is the concept DOI, which always resolves to the latest release and aggregates citations across all versions. To cite a specific version instead, use that version's DOI from the Zenodo record.
0.23.1.dev202609280633
Sep 28, 2026
0.23.1.dev202609270450
Sep 27, 2026
0.23.1.dev202609260412
Sep 26, 2026
0.23.1.dev202609250437
Sep 25, 2026
0.23.1.dev202609240431
Sep 24, 2026
0.23.1.dev202609230417
Sep 23, 2026
0.23.1.dev202609220422
Sep 22, 2026
0.23.1.dev202609210534
Sep 21, 2026
0.23.1.dev202609200406
Sep 20, 2026
0.23.1.dev202609190343
Sep 19, 2026
0.23.1.dev202609180359
Sep 18, 2026
0.23.1.dev202609170426
Sep 17, 2026
0.23.1.dev202609160417
Sep 16, 2026
0.23.1.dev202609150432
Sep 15, 2026
0.23.1.dev202609140527
Sep 14, 2026
0.23.1.dev202609130433
Sep 13, 2026
0.23.1.dev202609120330
Sep 12, 2026
0.23.0
Sep 12, 2026
0.22.1.dev202609110401
Sep 11, 2026
0.22.1.dev202609100403
Sep 10, 2026
0.22.1.dev202609090407
Sep 09, 2026
0.22.1.dev202609080402
Sep 08, 2026
0.22.1.dev202609070511
Sep 07, 2026
0.22.1.dev202609060343
Sep 06, 2026
0.22.1.dev202609050324
Sep 05, 2026
0.22.1.dev202609040402
Sep 04, 2026
0.22.1.dev202609030401
Sep 03, 2026
0.22.1.dev202609020403
Sep 02, 2026
0.22.1.dev202609010429
Sep 01, 2026
0.22.1.dev202608310622
Aug 31, 2026
0.22.1.dev202608300443
Aug 30, 2026
0.22.1.dev202608290535
Aug 29, 2026
0.22.1.dev202608281129
Aug 28, 2026
0.22.1.dev202608271032
Aug 27, 2026
0.22.1.dev202608252359
Aug 26, 2026
0.22.1.dev202608242358
Aug 25, 2026
0.22.1.dev202608240010
Aug 24, 2026
0.22.1.dev202608222352
Aug 23, 2026
0.22.1.dev202608212350
Aug 22, 2026
0.22.0
Aug 22, 2026
0.21.1.dev202608202357
Aug 21, 2026
0.21.1.dev202608192356
Aug 20, 2026
0.21.1.dev202608182354
Aug 19, 2026
0.21.1.dev202608172354
Aug 18, 2026
0.21.1.dev202608170007
Aug 17, 2026
0.21.1.dev202608152350
Aug 16, 2026
0.21.1.dev202608142348
Aug 15, 2026
0.21.1.dev202608140041
Aug 14, 2026
0.21.1.dev202608130044
Aug 13, 2026
0.21.1.dev202608120042
Aug 12, 2026
0.21.1.dev202608110018
Aug 11, 2026
0.21.0
Aug 10, 2026
0.20.1.dev202608100053
Aug 10, 2026
0.20.1.dev202608090006
Aug 09, 2026
0.20.1.dev202608080003
Aug 08, 2026
0.20.1.dev202608070026
Aug 07, 2026
0.20.1.dev202608060147
Aug 06, 2026
0.20.1.dev202608050146
Aug 05, 2026
0.20.1.dev202608040148
Aug 04, 2026
0.20.1.dev202608030259
Aug 03, 2026
0.20.1.dev202608020132
Aug 02, 2026
0.20.1.dev202608010130
Aug 01, 2026
0.20.1.dev202607310211
Jul 31, 2026
0.20.1.dev202607300139
Jul 30, 2026
0.20.1.dev202607290150
Jul 29, 2026
0.20.1.dev202607280146
Jul 28, 2026
0.20.1.dev202607270303
Jul 27, 2026
0.20.1.dev202607260135
Jul 26, 2026
0.20.1.dev202607250117
Jul 25, 2026
0.20.1.dev202607240136
Jul 24, 2026
0.20.1.dev202607230140
Jul 23, 2026
0.20.1.dev202607220138
Jul 22, 2026
0.20.1.dev202607210139
Jul 21, 2026
0.20.1.dev202607200228
Jul 20, 2026
0.20.1.dev202607190128
Jul 19, 2026
0.20.0
Jul 18, 2026
0.19.1.dev202607180103
Jul 18, 2026
0.19.1.dev202607170121
Jul 17, 2026
0.19.0
Jul 17, 2026
0.19.0.dev202607091635
Jul 09, 2026
0.18.1.dev202607160125
Jul 16, 2026
0.18.1.dev202607150126
Jul 15, 2026
0.18.1.dev202607140120
Jul 14, 2026
0.18.1.dev202607130240
Jul 13, 2026
0.18.1.dev202607120127
Jul 12, 2026
0.18.1.dev202607110106
Jul 11, 2026
0.18.1.dev202607101057
Jul 10, 2026
0.18.1.dev202607100248
Jul 10, 2026
0.18.1.dev202607091727
Jul 10, 2026
0.18.0
Jul 07, 2026
0.17.1
Jul 03, 2026
0.17.0
Jul 03, 2026
0.16.1
Jun 28, 2026
0.16.0
Jun 27, 2026
0.15.0
Jun 19, 2026
0.14.0
Jun 17, 2026
0.13.2
Jun 11, 2026
0.13.1
Jun 03, 2026
0.13.0
May 30, 2026
0.9.0
May 28, 2026
0.8.0
May 22, 2026
0.7.1
May 18, 2026
0.7.0
May 15, 2026
0.6.6
May 07, 2026
0.6.5
May 06, 2026
0.6.4
May 01, 2026
0.6.3
Apr 30, 2026
0.6.2
Apr 30, 2026
0.6.1
Apr 30, 2026
0.6.0
Apr 30, 2026
0.5.0
Apr 23, 2026
0.4.0
Apr 16, 2026
0.3.0
Mar 30, 2026
0.3.0a1
Mar 26, 2026
0.2.0
Mar 25, 2026
0.2.0a3
Mar 19, 2026
0.2.0a2
Mar 18, 2026
0.2.0a1
Mar 18, 2026
0.1.45
Feb 27, 2026
0.1.44
Feb 09, 2026
0.1.43
Jan 30, 2026
0.1.42
Jan 20, 2026
0.1.41
Jan 19, 2026
0.1.40
Jan 18, 2026
0.1.39
Jan 18, 2026
0.1.38
Jan 16, 2026
0.1.37
Jan 16, 2026
0.1.36
Jan 13, 2026
0.1.35
Jan 10, 2026
0.1.34
Jan 05, 2026
0.1.33
Jan 01, 2026
0.1.32
Dec 30, 2025
0.1.31
Dec 26, 2025
0.1.30
Dec 26, 2025
0.1.29
Dec 26, 2025
0.1.28
Dec 20, 2025
0.1.27
Dec 20, 2025
0.1.26
Dec 19, 2025
0.1.25
Dec 17, 2025
0.1.24
Dec 13, 2025
0.1.23
Dec 11, 2025
0.1.22
Dec 10, 2025
0.1.20
Dec 04, 2025
0.1.19
Dec 04, 2025
0.1.18
Nov 18, 2025
0.1.17
Nov 17, 2025
0.1.16
Nov 17, 2025
0.1.15
Nov 13, 2025
0.1.14
Nov 12, 2025
0.1.13
Nov 12, 2025
0.1.12
Nov 10, 2025
0.1.11
Nov 09, 2025
0.1.10
Nov 08, 2025
0.1.8
Nov 08, 2025
0.1.7
Nov 08, 2025
0.1.6
Nov 08, 2025
0.1.5
Nov 08, 2025
0.1.4
Nov 07, 2025
0.1.3
Nov 07, 2025
0.1.2
Nov 06, 2025
0.1.1
Nov 05, 2025
0.1.0
Nov 02, 2025
Wheel compatibility matrix
Files in release
Extras:
Dependencies:
pydantic
(>=2.12.0)
shortuuid
(>=1.0.13)
typer
(>=0.16.0)
requests
(>=2.32.4)
pyyaml
(>=6.0.2)
rich
(>=14.1.0)
toml
(>=0.10.2)
tenacity
(>=9.1.2)
python-dotenv
(>=1.1.1)
litellm
(>=1.92.0)
jinja2
(>=3.1.6)
dirhash
(>=0.5.0)
packaging
(>=25.0)
fastapi
(>=0.128.0)
uvicorn
(>=0.38.0)
pathspec
(>=1.0.3)
supabase
(>=2.28.2)
httpx
(>=0.27.0)
filelock
(>=3.29.4)
platformdirs
(>=4.10.0)
pyjwt
(>=2.10.1)
jsonschema
(>=4.23.0)