What is Harbor
Harbor evaluates and optimizes agents and language models inside container environments. It comes from the creators of Terminal-Bench.
Four things it does:
- Evaluate arbitrary agents (Claude Code, Codex CLI, OpenHands, and others)
- Build and share your own benchmarks and environments
- Run thousands of environments in parallel via Daytona, Modal, LangSmith, Blaxel, Novita Sandbox, Tensorlake, Runta, and Sail
- Generate rollouts for RL optimization
The life of a trial
Section titled “The life of a trial”Task ──► start env ──► agent ──► Trajectory │ │ └──► stop env ──► Verifier ──► Reward ├─► aggregate evals └─► RL / SFTThe trajectory and the reward are separate artifacts. Trajectories feed SFT and failure analysis; rewards feed aggregate metrics and optimization. That is why trajectory audit and task verification are different products.
Why “verifiable”
Section titled “Why “verifiable””- The reward is produced by code in
tests/test.sh, not by a model’s opinion - A task is reviewable in a pull request: instruction, environment, tests are files
- The same task should give the same verdict across providers and machines
How it differs from other harnesses
Section titled “How it differs from other harnesses”| Emphasis | |
|---|---|
| Harbor | Task-level verifiable reward, container environments, massive parallelism, RL rollouts |
| One-off benchmark scripts | Single run on a fixed dataset, hard to extend |
| Pure LLM-as-judge | No execution check, reward not reproducible |