Task eligibility

Galapagos is a platform for research-grade discovery and optimization tasks, not merely any program that can be executed and graded. A new task is eligible for the research catalog and leaderboard only when it satisfies all three requirements below.

All three are required

A runnable seed, a card.yaml, and an evaluator make a task executable. They do not by themselves make it a Galapagos research task. The problem must be unsolved, open-ended, and deterministically verifiable.

1. Unsolved and genuinely challenging

No known reproducible solution has achieved the task's documented perfect score under the exact submitted protocol and resource budget.

The submission must:

• define what a perfect score means, including the metric direction and any ceiling or target;
• report the best known score, its source, and the date or upstream revision used;
• show a meaningful gap between that result and perfection; and
• start from a credible baseline, so progress is not just the recovery of an obvious missing feature.

If the metric has no finite ceiling, the author must instead document the best known result and why the objective still has demonstrable headroom. Once a reproducible perfect solution exists, the task is saturated: keep its historical results, but retire it from active discovery or release a new version with harder instances, a stronger baseline, or a tighter resource budget.

2. Open-ended research and optimization

The task must reward meaningful improvements to an algorithm, model, design, system, or scientific hypothesis. There should be multiple plausible approaches and no prescribed sequence of edits that turns the task into a hidden tutorial.

Its primary metric must provide continuous or finely graded feedback across candidate quality. A binary validity check is useful as a gate, but pass/fail alone is not a discovery objective. Better ideas should be able to earn better scores repeatedly under a fixed evaluation budget.

Good task documentation identifies the available improvement axes—for example model architecture, search policy, approximation quality, sample efficiency, runtime, memory use, or physical design—and fixes the compute, query, data, and wall-clock budgets that make comparisons fair.

3. Deterministically verifiable

A trusted evaluator must independently recompute validity and score from the candidate's raw output. For the same candidate, fixtures, seed, dependency versions, and hardware contract, evaluation must produce the same verdict and score, subject only to an explicitly documented numerical tolerance.

In particular:

• never trust a score, loss, runtime, or success flag reported by the candidate itself;
• pin datasets, test cases, random seeds, dependencies, and the evaluation environment;
• keep scorer code and held-out answers outside the candidate's writable boundary;
• turn stochastic evaluation into a fixed-seed aggregate with a documented rule;
• document and control any unavoidable numerical or timing noise; and
• do not use human opinion, an LLM judge, a live mutable API, or an unrepeatable external experiment as the sole source of the score.

An LLM or human review may provide a secondary policy or anti-cheating audit. The leaderboard metric itself must still come from a reproducible, task-native verifier.

Admission checklist

A task submission should provide evidence for every item:

• Unsolved: No known solution has reached the defined perfect score.
• Headroom: The best known score and remaining gap are documented.
• Open-ended: Multiple research or optimization approaches can improve the candidate.
• Graded: The primary score distinguishes partial progress rather than only pass/fail.
• Independent: The evaluator recomputes the score instead of accepting candidate claims.
• Reproducible: Data, seeds, dependencies, budgets, and tolerances are pinned.
• Protected: Evaluator assets and held-out answers are not candidate-writable.

Task pull requests record this evidence in the task-submission template. The automated bundle gate checks portability and evaluator execution; maintainers review the unsolved and open-ended claims because those properties cannot be established by schema validation alone.

What does not qualify

The following can still be useful as tutorials, smoke tests, or internal fixtures, but they are not eligible as research-grade leaderboard tasks:

• a benchmark with a known reproducible perfect solution;
• a fixed-answer exercise whose only metric is pass/fail;
• a task that rewards copying a public answer or lookup table;
• a scorer based only on subjective human or LLM preference;
• a candidate that reports its own score without independent recomputation; or
• an evaluation that changes with an unpinned API, dataset, random seed, or environment.

Demo and CI fixtures should be labeled as such and kept separate from claims about open-ended research performance.