Leaderboard Submission: Muse Spark 1.1 (xhigh) + mini-SWE-agent

[github-actions]

Promoted from #93 (author @alexgshaw, head a67237e).
Trials are cloned into leaderboard-owned copies; the checks below ran against the cloned trials.

Maintainer comments /judge then /apply to recompute metrics with disqualified trajectories.

---

✅ Static Analysis [PASS]

9 checks passed

Check	Result
Valid source_filter	✅
Valid metadata	✅
timeout_multiplier default (None or 1.0)	✅
No agent timeout overrides	✅
No verifier timeout overrides	✅
No resource overrides	✅
Valid trial count (89 tasks × ≥5)	✅
Valid per-trial records	✅
Valid task digests	✅

Trial Summary

Result	Count
No error	390
AgentTimeoutError	53
NonZeroAgentExitCodeError	1
NetworkConnectionError	1
Total	445

Submission Summary

Model [org]	Effort	Agent [org]	Date	Accuracy	± SE	Tokens	Cost
Muse Spark 1.1 [Meta]	xhigh	mini-SWE-agent [Princeton]	2026-07-09	76.18%	1.23%	945,980,322	$198.05

_{Ran on a67237e. Errored trials count reward 0 in the metric computation.}

[alexgshaw]

/judge

[github-actions]

✅ Static Analysis [PASS]

9 checks passed

Check	Result
Valid source_filter	✅
Valid metadata	✅
timeout_multiplier default (None or 1.0)	✅
No agent timeout overrides	✅
No verifier timeout overrides	✅
No resource overrides	✅
Valid trial count (89 tasks × ≥5)	✅
Valid per-trial records	✅
Valid task digests	✅

Trial Summary

Result	Count
No error	390
AgentTimeoutError	53
NonZeroAgentExitCodeError	1
NetworkConnectionError	1
Total	445

Submission Summary

Model [org]	Effort	Agent [org]	Date	Accuracy	± SE	Tokens	Cost
Muse Spark 1.1 [Meta]	xhigh	mini-SWE-agent [Princeton]	2026-07-09	76.18%	1.23%	945,980,322	$198.05

_{Ran on 544b081. Errored trials count reward 0 in the metric computation.}

[github-actions]

Harness cheating 🔴 [maintainer reviews - renders a submission invalid]

• d059c5e2 · code-from-image · judge verdict
• b2076610 · code-from-image · judge verdict
• 96d3a029 · fix-code-vulnerability · judge verdict
• 40b49c2b · code-from-image · judge verdict
• b705c8e0 · code-from-image · judge verdict
• 5b1ac706 · code-from-image · judge verdict

Reward hacking 🟡 [submitter comments on any false positive - maintainer can uncheck and /apply]

• b142a990 · db-wal-recovery · judge verdict
• d2d16cad · compile-compcert · judge verdict
• c66b4d3e · path-tracing · judge verdict
• 3bd7bc5b · large-scale-text-editing · judge verdict
• 8ea2542a · extract-elf · judge verdict
• 6ad4cce0 · path-tracing · judge verdict
• 99928989 · git-multibranch · judge verdict
• b4ee40dc · path-tracing · judge verdict
• 30e685df · path-tracing · judge verdict
• b6b18ea9 · model-extraction-relu-logits · judge verdict
• 3e789041 · fix-code-vulnerability · judge verdict
• 52043221 · db-wal-recovery · judge verdict
• 8d621d71 · extract-elf · judge verdict

Refusals ⚪ [for information - sometimes a cause of agent failure]

• none

No Trajectory ⚫ [infra error - no trajectory to judge]

• none

_{Ran on 61c0453. 339 trajectories judged, 19 flagged. Judge job: Job. ✅ 0 disqualified trials applied at ab64cca.}

[alexgshaw]

/apply

[github-actions]

Applied Disqualifications — no reward-hacking trials checked

Accuracy 76.18% (± 1.23%) → 76.18% (± 1.23%)

Submission Summary

Model [org]	Effort	Agent [org]	Date	Accuracy	± SE	Tokens	Cost
Muse Spark 1.1 [Meta]	xhigh	mini-SWE-agent [Princeton]	2026-07-09	76.18%	1.23%	945,980,322	$198.05

_{Ran on ab64cca. Submission json updated with disqualified_trials and updated metrics.}

[github-actions]

✅ Entry submitted to the leaderboard

Model [org]	Effort	Agent [org]	Accuracy	Tokens	Cost	PR	Anti-Cheat
Muse Spark 1.1 [Meta]	xhigh	mini-SWE-agent [Princeton]	76.2% ± 1.2%	945,980,322	$198.05	#94	Summary

_{Ran on dde3cd9.}