Hi @joe-needham and thanks for responding to my comment on issue #124 earlier. After you separated the two submissions in PR #130, I got curious and went through the published source code and tech reports for every submission I could find.

What jumped out: at least two other agents on the main board verifiably use test-set information to guide their runs. The teams that disclosed their methodology (or self-reported it) got penalized, while teams that didn’t are assumed compliant. Furthermore, the main board isn’t an apples-to-apples comparison to begin with.

Here’s what I found.

The main leaderboard is not directly comparable to begin with

The separation of Disarray and LoongFlow into “Additional Submissions” implies that the remaining main leaderboard is an apples-to-apples comparison. It isn’t. The main board has uncontrolled variation across every dimension that affects performance:

Hardware. Submissions range from a single V100 (12 vCPUs, 220 GB RAM) to dual H100s (48 vCPUs, 220 GB RAM). An H100 delivers roughly 3.4x the FP16 throughput of an A100, so agents on H100 hardware can train models and explore solutions over 3x faster within the same wall-clock budget. Dual-GPU setups double that again. CAIR MARS+ on 2x H100 has up to 7x the GPU compute throughput of R&D-Agent on a V100 — yet both sit on the same leaderboard with no distinction. Famou-Agent uses 64 vCPUs and 500 GB RAM — nearly 3x the CPU and over 2x the memory of the recommended spec.

Runtime. The leaderboard spans 12 to 36 hours — a 3x range. Compounded with hardware differences, the effective compute gap is even larger.

External knowledge. Some submissions use web search at runtime to retrieve SOTA models and competition-winning solutions (MLE-STAR, KAPSO, MARS+), while others ship pre-populated knowledge bases with benchmark-specific model recommendations (MLEvolve, ML-Master 2.0). These sources can significantly bootstrap the search process. There is no disclosure requirement or distinction on the leaderboard.

Test-set feedback methodology. This is the dimension used to justify the separation — but it is one uncontrolled dimension among many. Without ablation studies, there is no basis for concluding that test-set feedback has more impact on scores than a 7x hardware advantage or access to ground-truth external knowledge.

Other main-board submissions verifiably use test-set feedback

How test-set feedback is used matters here. Continuous hill-climbing, where the test score directly guides the search, is a fundamentally different practice than using a binary status flag.

FM-Agent / Famou-Agent (main leaderboard — three entries: 43.56%, 59.56%, and 64.44%)

FM-Agent holds three entries on the main leaderboard, including the current #1 position (64.44%). All three link to the same repository (baidubce/FM-Agent). That repository contains a single MLE-bench evaluator that uses the test score as an evolutionary fitness signal.
The complete pipeline in
examples/MLE-insulting/evaluator.py:

1. run_grader()
   calls mlebench grade-sample (private test-set grading) → returns the test score + medal status
2. scale_score_with_medal()
   computes sigmoid(test_score) + medal_bonus (+1 bronze, +2 silver, +3 gold)
3. evaluate()
   returns this combined_score as the evolutionary fitness for every candidate

# evaluator.py, lines 377-397
def scale_score_with_medal(raw_score, medal: str = "none"):
    score = -raw_score if lower_is_better else raw_score
    combined_score = float(1 / (1 + np.exp(-score)))
    if medal == "none":
        return combined_score
    elif medal == "bronze":
        return combined_score + 1.0
    elif medal == "silver":
        return combined_score + 2.0
    elif medal == "gold":
        return combined_score + 3.0

Also:

• No validation-set scoring path exists anywhere in the repository’s 86-commit history — the only MLE-bench evaluator grades against the test set.
• The evolved program (best_program.py) computes no validation metrics — it only generates test predictions.

On PR #123, the Famou-Agent team stated their agent uses “feedback solely from an independently partitioned validation set.” The repository was subsequently wiped on March 15, 2026 — after test-set feedback concerns were raised in February — though the code remains accessible via earlier commits and forks. No updated code supporting the validation-set claim has been published.
None of the three FM-Agent entries was moved to the separate leaderboard.

KAPSO / Leeroo (main leaderboard, 50.67%)

KAPSO receives the full grading result on every submission and uses the medal status for early termination. The source code at benchmarks/mle/handler.py shows:

• Line 213: grade_csv(submission_path, competition) — full test-set grading runs on every valid submission
• Line 220: self.got_medal = self.got_medal or grading_results.any_medal
• Lines 260–261: def stop_condition(self): return self.got_medal — test-set medal controls early termination

The KAPSO paper (arXiv:2601.21526, Section 5.1) openly discloses this: “Stop early if the run achieves any medal according to the MLE-Bench grading library.” KAPSO was not moved to the separate leaderboard.

The separation does not track severity

If we look at how much test-set information is actually used across these submissions, the current policy doesn’t track severity at all:

FM-Agent and LoongFlow both verifiably use continuous test-set scores as evolutionary fitness — yet FM-Agent remains on the main leaderboard while LoongFlow was separated. KAPSO uses medal-based early stopping, yet remains on the main board while Disarray was separated merely for self-reporting feedback usage without a verifiable codebase.
I am pointing this out to show that maintaining a consistent separation on this dimension requires auditing every submission, and the evidence above shows how impractical that is.

The verification gap makes consistent enforcement impractical

Of the 28 main-board entries, only 15 publish a complete implementation that allows independent verification of test-set feedback methodology. Another 8 provide partial artifacts, and 5 have no published code or tech reports at all.
Under the current policy, unverifiable submissions are treated as compliant by default, while submissions that transparently disclosed their approach were separated. This creates a perverse incentive structure: teams that publish less code and disclose less methodology are less likely to be flagged.

Suggestion: Return to one board, use transparency columns

The leaderboard already handles differences in Runtime via transparency columns. If runtime and models used already belong next to the score, why not hardware, external knowledge, and test-set feedback? These all change how a result should be read.
I took a pass at compiling these dimensions across all submissions from published code, papers, and submission PRs to do the legwork here:

I’d restore a single leaderboard and keep these transparency columns visible as first-class fields next to performance. That gives readers the full picture in one view and avoids the maintenance burden of policing a separation that, as shown above, is virtually impossible to enforce consistently anyway.
Happy to be corrected on any of the entries. Thanks!