Community Leaderboard

Leaderboard

Community-submitted runs on the 46-task Long-Horizon Terminal-Bench suite, ranked by mean reward. The seed entries are our Terminus-2 baselines — the same runs analyzed in the report. LHTB is open and follows the same submission flow as Terminal-Bench 2: run all 46 tasks once under the same 90-minute budget, then open a pull request with your job directories. A bot validates the submission and a maintainer reviews the trajectories against the hidden verifiers before it lands on the board. A verified badge means we reproduced it.

Verified only

Rank	Agent	Model	Submitter	Date	Solved	Mean reward

Scores are mean reward over the 46-task suite (errors = 0; a task counts as solved at reward ≥ 0.95). The seed entries are our Terminus-2 baselines; the board grows as the community submits.

Submit a run

Same flow as Terminal-Bench 2 — run all 46 tasks once, then open a pull request.

Using an AI coding agent? Point it at our agent skill so your agent can run the suite, build metadata.yaml, and open the submission pull request for you.

1
Run all 46 tasks once
Evaluate a model · reference harness
harbor run -d long-horizon-terminal-bench \ -a terminus-2 -m "your-provider/your-model" -k 1
Bring your own agent
harbor run -d long-horizon-terminal-bench \ --agent-import-path "path.to.agent:YourAgent" -k 1
Set the 90-minute cap with override_timeout_sec: 5400 and timeout_multiplier: 1.0 on the agent (matching the reference configs) so your run is comparable to the board.

Lay out your jobs — no fork needed

The LHTB leaderboard is a Hugging Face Dataset, so there is no fork — a PR pushes to a refs/pr/N branch directly on the repo (step 3). Just arrange your job folder(s) locally under submissions/long-horizon-terminal-bench/1.0/<agent>__<model>/ alongside a metadata.yaml.

Submission layout

submissions/long-horizon-terminal-bench/1.0/
  terminus-2__your-model/
    metadata.yaml
      <job-folder>/
      config.json
      <trial-1>/result.json   # 1 trial/task, with artifacts

metadata.yaml

agent: terminus-2
model: your-provider/your-model
reasoning_effort: default             # exact tier used: minimal/low/medium/high, adaptive, off, or "default"
organization: Your Org
repo: https://github.com/your/agent   # optional
docs: https://.../model-card          # optional

3
Open a pull request — no fork

The Hugging Face CLI uploads your folder to a fresh refs/pr/N branch and opens the PR in one step (browser alternative: Community → New pull request on the dataset).
Upload + open PR · no fork
pip install -U huggingface_hub && hf auth login hf upload IntelligenceLab/LHTB-leaderboard \ submissions/long-horizon-terminal-bench/1.0/terminus-2__your-model \ submissions/long-horizon-terminal-bench/1.0/terminus-2__your-model \ --repo-type dataset --create-pr
Our bot auto-validates the submission and comments on any errors. Once it passes, a maintainer reviews the trajectories and merges — then your run is imported to the board with a verified badge.

Validation rules. Uniform 90-minute agent budget per task (override_timeout_sec = 5400, the shared budget every leaderboard baseline used) · timeout_multiplier = 1.0 · one trial per task (-k 1) · tasks flagged continue_until_timeout keep iterating within the 90-minute window · no CPU / memory / storage overrides · every trial has a valid result.json with run artifacts · agents may not access the LHTB website or repository (reward hacking). Errored trials score 0.

Want the full analysis behind these baselines — how the tasks are built, how they're graded, and where today's frontier runs out of steam? Read the blog →