Skip to content

What is Harbor

Harbor evaluates and optimizes agents and language models inside container environments. It comes from the creators of Terminal-Bench.

Four things it does:

  1. Evaluate arbitrary agents (Claude Code, Codex CLI, OpenHands, and others)
  2. Build and share your own benchmarks and environments
  3. Run thousands of environments in parallel via Daytona, Modal, LangSmith, Blaxel, Novita Sandbox, Tensorlake, Runta, and Sail
  4. Generate rollouts for RL optimization
Task ──► start env ──► agent ──► Trajectory
│ │
└──► stop env ──► Verifier ──► Reward
├─► aggregate evals
└─► RL / SFT

The trajectory and the reward are separate artifacts. Trajectories feed SFT and failure analysis; rewards feed aggregate metrics and optimization. That is why trajectory audit and task verification are different products.

  • The reward is produced by code in tests/test.sh, not by a model’s opinion
  • A task is reviewable in a pull request: instruction, environment, tests are files
  • The same task should give the same verdict across providers and machines
Emphasis
Harbor Task-level verifiable reward, container environments, massive parallelism, RL rollouts
One-off benchmark scripts Single run on a fixed dataset, hard to extend
Pure LLM-as-judge No execution check, reward not reproducible