Why Galapagos¶
LLM-driven evolutionary search repeatedly asks a model for a candidate, scores it, and uses the result to decide what to try next. The idea is simple; comparing implementations is not. Each method usually arrives with its own population format, prompt assembly, evaluator contract, environment, and run log.
The comparison problem¶
Two reported scores can differ for reasons that have little to do with the search algorithm:
- different task versions or seeds;
- different evaluator dependencies and hardware;
- different candidate-validity rules;
- different iteration and retry accounting;
- different model, prompt, or sampling settings;
- different handling of checkpoints and failed candidates.
Without a shared contract it is hard to tell whether one search policy is better, merely tuned differently, or evaluated under a different benchmark.
The Galapagos approach¶
Galapagos makes the comparison surface explicit:
- Six component roles. Population, SelectionPolicy, PromptBuilder, Proposer, Evaluator, and Memory describe the moving parts implemented by the current scaffold catalog.
- Task-owned evaluation. A task card carries its seed, evaluator, metrics, and environment. Compatible scaffolds therefore use the same scorer source for that task.
- Run-level execution.
general.eval_modechooses Docker or local evaluation. Docker is the reproducible default; local mode is an explicit operator choice. - Typed config and receipts. Effective config, model accounting, trajectories, checkpoints, and execution metadata make important differences visible.
- One catalog protocol. Scaffold and task cards can be loaded locally and submitted to the Hub.
What is implemented today¶
The bundled runnable scaffold catalog contains exactly fourteen controllers:
| Family | Scaffolds |
|---|---|
| Evolutionary methods | openevolve, adaevolve, evox, gepa |
| Search baselines | beam_search, best_of_n, best_of_n_attempts, random, topk |
| Algorithm-engineering agents | ale_agent, algotune_agent |
| Agent/meta baselines | claude_code, codex, meta_harness |
OpenEvolve supplies island-model MAP-Elites search. AdaEvolve adds adaptive exploration, UCB island allocation, and stagnation tactics. EvoX evolves and hot-swaps the sampling strategy. The five search baselines provide controlled comparisons. ALE-Agent contributes domain-guided best-first algorithm-engineering search; AlgoTune Agent contributes a command-driven, mutable optimization workspace with evaluation and profiling tools. Claude Code and Codex each delegate a whole run to one subscription-authenticated CLI agent. Meta-Harness delegates selection and generation to a skill-steered coding agent (or an explicit API-LLM fallback) over an append-only archive.
The task catalog currently contains 1,274 task cards. 1,269 are runnable; five single-cell
cards are spec entries without a seed/evaluator pair. Hardware and dependency requirements still
matter: a runnable card is not a promise that every host can execute it.
Methods without a registered controller are not listed as Galapagos capabilities. A community card
with status: spec can describe a design on the Hub, but it does not become runnable until it ships
loadable component code or a registered GalapagosScaffold controller.
What the Hub records¶
The Hub stores scaffold, task, and discovery metadata and artifacts, and exposes review states such
as pending, accepted, and rejected. Accepting a discovery promotes it to the leaderboard in the
current backend. The backend does not automatically replay a trajectory or re-score a solution,
so the review state should not be described as an automated verification guarantee.
Port fidelity is handled separately by the repository's executable upstream-review workflow: a
reviewer contributes a test under porting_reviews/, and CI reruns that test when the port changes.
The practical goal¶
Galapagos does not make every method or task interchangeable. It makes incompatibilities explicit and gives supported scaffold/task pairings the same API, config vocabulary, evaluator boundary, and artifact format. That is the foundation needed for comparisons that can be inspected and repeated.