Verifier and reward
Entrypoint
Section titled “Entrypoint”tests/test.sh (tests/test.bat on Windows) is the verifier entrypoint. After the
agent stops, Harbor uploads tests/ to /tests/ and runs it.
It must write a reward
Section titled “It must write a reward”| File | Format |
|---|---|
/logs/verifier/reward.txt |
a single number, usually 1 or 0 |
/logs/verifier/reward.json |
object of labeled numeric metrics |
If both exist, Harbor reads reward.json.
#!/bin/bashset -euo pipefail
if pytest /tests/test_outputs.py; then echo 1 > /logs/verifier/reward.txtelse echo 0 > /logs/verifier/reward.txtfiThe upstream hello-world example does the same, plus a CTRF report:
uvx --with pytest==8.4.1 --with pytest-json-ctrf==0.3.5 \ pytest --ctrf /logs/verifier/ctrf.json /tests/test_state.py -rAShared vs separate verification
Section titled “Shared vs separate verification”| Mode | Behavior |
|---|---|
shared (default) |
Verifier runs in the agent environment; test deps must be in the agent image |
separate |
Use [verifier.environment], or a Dockerfile in tests/ |
In separate mode Harbor does not upload tests into the verifier image; the
image must already contain /tests/test.sh. Agent filesystem changes are not
inherited, so declare what you need in artifacts.
Known traps
Section titled “Known traps”- Reward hacking — if the test can be bypassed, agents learn to bypass it. The
repo ships a
reward-hackingexample - Reliability — use
set -euo pipefailand absolute paths; cwd surprises are real - Multiple metrics — use
reward.json; do not collapse them into one weighted number