Task structure
Layout
Section titled “Layout”my-task/├── instruction.md # what the agent is told├── task.toml # timeouts, resources, verifier config├── environment/│ ├── Dockerfile # docker-compose.yaml and Apptainer.def also supported│ └── ...├── solution/│ └── solve.sh # reference solution, used for oracle runs└── tests/ ├── test.sh # verifier entrypoint └── ...instruction.md
Section titled “instruction.md”Only what the agent should see. Never the grading logic. From the upstream
hello-world example:
Create a file called hello.txt with "Hello, world!" as the content.task.toml
Section titled “task.toml”Real fields (schema_version = "1.4"):
schema_version = "1.4"
[task]name = "harbor/hello-world"version = "1.0.0"authors = []keywords = []
[metadata]difficulty = "easy"category = "programming"tags = ["trivial"]
[verifier]timeout_sec = 120.0
[agent]timeout_sec = 120.0
[environment]build_timeout_sec = 600.0cpus = 1memory_mb = 2048storage_mb = 10240gpus = 0mcp_servers = []
[verifier.env]
[solution.env]Verifiability checklist
Section titled “Verifiability checklist”- The instruction only relies on what exists in the environment
-
tests/can decide success without a human -
solution/solve.shscores full marks (oracle baseline) - There is no shortcut around the tests (reward hacking)
- Resources and timeouts are pinned in
task.toml