Skip to content

Verifier and reward

tests/test.sh (tests/test.bat on Windows) is the verifier entrypoint. After the agent stops, Harbor uploads tests/ to /tests/ and runs it.

File Format
/logs/verifier/reward.txt a single number, usually 1 or 0
/logs/verifier/reward.json object of labeled numeric metrics

If both exist, Harbor reads reward.json.

tests/test.sh
#!/bin/bash
set -euo pipefail
if pytest /tests/test_outputs.py; then
echo 1 > /logs/verifier/reward.txt
else
echo 0 > /logs/verifier/reward.txt
fi

The upstream hello-world example does the same, plus a CTRF report:

终端窗口
uvx --with pytest==8.4.1 --with pytest-json-ctrf==0.3.5 \
pytest --ctrf /logs/verifier/ctrf.json /tests/test_state.py -rA
Mode Behavior
shared (default) Verifier runs in the agent environment; test deps must be in the agent image
separate Use [verifier.environment], or a Dockerfile in tests/

In separate mode Harbor does not upload tests into the verifier image; the image must already contain /tests/test.sh. Agent filesystem changes are not inherited, so declare what you need in artifacts.

  • Reward hacking — if the test can be bypassed, agents learn to bypass it. The repo ships a reward-hacking example
  • Reliability — use set -euo pipefail and absolute paths; cwd surprises are real
  • Multiple metrics — use reward.json; do not collapse them into one weighted number