Tasks

A task is the evaluation problem a scaffold is run against. The catalog ships 1,274 task cards; 1,269 are runnable and five are spec entries without a complete seed/evaluator pair. Each is a folder with a card.yaml under src/galapagos/tasks/<name>/. List them at runtime:

import galapagos as gx
gx.available_tasks()    # all 1,274 bundled task cards (runnable + spec)

galapagos task list

Catalog

The 1,274 task cards span open mathematics, algorithm and scientific-computing speedups, systems research, GPU kernels, AtCoder heuristic contests, FrontierCS problems, AlgoTune tasks, OpenProblems tasks, BioML-Bench/AutoScientists biomedical ML, scientific equation discovery (the 240-problem LLM-SRBench suite), and prompt/ML benchmarks. The five spec cards are cell_cell_communication, differential_expression, foundation_models, grn_inference, and ist_preprocessing. Runnable families can still require their domain-specific dependency stack (GPU tooling, judge packages, scientific libraries, or API keys) to produce meaningful scores.

The two canonical quickstart tasks:

Name	Display	Domain	Status	Metric	Summary
circle_packing	Circle Packing (n=26)	math	stable	combined_score (maximize)	Pack 26 circles in the unit square; maximize the sum of radii.
function_minimization	Function Minimization	math	stable	combined_score (maximize)	Find (x, y) minimizing f(x,y) = sin(x)·cos(y) + sin(x·y) + (x² + y²)/20.

Both are pure-Python, single-file, EVOLVE-BLOCK tasks with no GPU requirement. They are scored in Docker by default, so a first run wants a running Docker daemon (or --general.eval_mode local to score in a host subprocess instead). function_minimization is the smallest — the recommended task for a quick first run.

Notes per task

• circle_packing — the classic AlphaEvolve benchmark. You evolve construct_packing() inside the EVOLVE-BLOCK; the fixed entry point run_packing() -> (centers, radii, sum_radii) calls it. The score and validity are recomputed independently from the returned geometry (anti reward-hacking). Best known ≈ 2.635.
• function_minimization — a non-convex 2-D objective with a central basin. You evolve the search inside the EVOLVE-BLOCK (search_algorithm); the fixed entry point run_search() returns (x, y) or (x, y, value) (the bundled seed returns the 3-tuple), and the score rises as f decreases. Approximate global minimum ≈ −1.519, at (−1.704, 0.678). Instant — the task to reach for on a first run. (The Hub Playground runs circle_packing.)

The AlgoTune family (algotune_*, 154 tasks)

AlgoTune scores correctness plus a stopwatch, and that changes what the evaluator has to defend against.

The seed you start from is the reference implementation — AlgoTune's, inlined verbatim — so it scores exactly 1.0x and your job is to make it faster without breaking it. The score is the speedup, sum(reference_ms) / sum(your_ms), measured over 10 problem instances of the size at which the reference takes about 100 ms (AlgoTune's own per-task n). Correctness is recomputed from your output by the task's own is_solution, and one wrong answer voids the speedup entirely rather than being averaged away — as it does upstream. No instance may take more than 10× the reference.

One instance in a while is simply dropped. A few of AlgoTune's validators are quality gates rather than equality checks (spectral_clustering's scores a clustering instead of comparing it), and the reference does not always clear its own bar — it is rejected on roughly 1 instance in 40. Upstream never notices, because it only ever validates the solver; here the seed is the reference, so an unlucky draw would otherwise void a perfect candidate for failing a test the ground truth also fails. You are asked to match the reference and beat its clock, so an instance the reference cannot pass is not a fair test of you.

The environment is AlgoTune's own — its published requirements.txt, shipped as a dedicated image (galapagos-task-base-algotune). That means the optimisation toolbox (numba, Cython, pythran, dace, JAX, torch-CPU, dask, cvxpy, OR-Tools, scikit-learn, …) for every task in the family, not just the ones whose reference happens to import them — reaching for it is the point, and upstream's own published solutions get their speedups with numba (469 of its 2831 result files) and Cython (290). It also means AlgoTune's numpy==1.26.4: lqr and delaunay are written against numpy 1.x APIs that numpy 2 removed, so on any other image they score a permanent zero with upstream's code byte-for-byte intact. Each card's system_message lists what is installed, and work done at import time (a JIT, a Cython build) is not charged to your runtime.

Three shortcuts are closed, and the harness will not reward them

AlgoTune measures one thing, so anything that makes the clock read low wins — whether or not any work was done. Upstream is safe because it scores on a test split its agent never sees and times each instance in a fresh subprocess. galapagos has neither, so these are closed structurally instead:

• Caching a problem you have already solved buys nothing: every instance is timed exactly once, cold, after a warm-up on an instance that is never scored.
• Precomputing the scored answers at import has nothing to precompute: the instances are drawn from random seeds after your module is imported and warmed up.
• Doing all the work in one call and coasting through the rest does not move a ratio of totals.

One thing is not closed. Your code shares a process with the scorer, so it can reach into sys.modules and patch the clock. Nothing in-process can prevent that — it is why AlgoTune uses a subprocess, and why galapagos's evaluate(program_path) contract cannot copy that. Treat leaderboard timing claims as review-sensitive rather than tamper-proof.

Because the instances are drawn afresh, the score is a measurement, not a constant: re-scoring the same program moves it by about ±2%, on top of the wall-clock noise any timing benchmark carries.

Working with a task

import galapagos as gx
task = gx.GalapagosTask.from_card(name="circle_packing")   # bundled -> cache -> Hub
# An explicit path is always local: path="tasks/circle_packing/card.yaml"

task.context           # the problem statement injected into prompts
task.status            # 'stable'
task.runnable          # True iff it ships a seed + evaluator.py
task.initial_genome()  # the seed Genome (content = the seed program)
task.evaluator         # the task's Evaluator: a ContainerEvaluator (the default — the task's own
                       # image) or a SubprocessEvaluator (general.eval_mode: local)

# score any candidate directly:
seed = task.initial_genome()
task.evaluator.evaluate(seed).combined_score

task.set_eval_mode("docker")      # how general.eval_mode reaches the task

Docker vs. local. By default the task's evaluator.py runs inside the task's own container image: the same evaluator source in the task-owned environment (see Task environments for how that image is resolved and built). Where it runs is a run-level choice and the card has no say in it: general.eval_mode, which galapagos run --general.eval_mode local sets and task.set_eval_mode() sets from code. It is the only input, so a run is reproducible from its config alone. (Exactly one task cannot run in a container at all: mlx_metal_kernel_opt, which is Apple-Metal-only and therefore the one task that ships no Dockerfile. Score it with --general.eval_mode local.) See Write your own task → Docker evaluation.

Card fields

Each task card (a TaskCard) records: name, display_name, organization (the Hub repo id is <organization>/<name>), domain, family, summary, system_message, metrics (a list of {metric_name, metric_direction, metric_description, metric_computation} — the legacy single metric {key, direction, type} is still accepted), components ({initial_program, evaluator} file pointers — runnable cards must declare initial_program, and its extension must match the canonical language; an optional evaluate_final function lives in the same evaluator file rather than a second component), environment ({est_runtime_s, gpu, api_keys, env, network, docker_access, cascade_evaluation, cascade_thresholds, dockerfile} plus the prose requires / toolchain — every key the loop actually reads, and there is no mode among them. It replaces the old evaluation + constraint pair, which described the same thing and had begun to collide; an un-migrated card still folds), language (required canonical value such as python or cpp), modality, assets, external_resources (large data fetched at load time), references, and metadata. There is no seed block: it described the shape of the seed ({kind, language, evolve_blocks}) and no code read one of those keys, so they drifted — one card claimed evolve_blocks: true over a seed with no marker in it. components.initial_program names the seed; what is IN that file is a question for the file.

from galapagos.cards.registry import load_task_card
card = load_task_card("circle_packing")
card.metric.key        # 'combined_score'
card.metric.direction  # 'maximize'
card.components        # {'initial_program': 'initial_program.py', 'evaluator': 'evaluator.py'}

The system_message becomes task.context and is sent as the proposer's system message, replacing the scaffold's generic default. Each task directory also carries a README.md — the human-facing write-up (what the problem is, why it matters, how it is scored, how to run it). The metric block declares the headline number and direction; the leaderboard displays submitted scores and their review state by task.

To add a task, see Write your own task.