Write your own task¶

Check research eligibility first

A research-grade Galapagos task must be unsolved, open-ended, and deterministically verifiable. All three are required; an executable seed and scorer are not sufficient on their own. Read the task eligibility policy before building the bundle.

A task is normally five files in a directory under tasks/<name>/, and the submission gate requires every one of them. A genuinely host-only task follows the explicit metadata exception documented there:

tasks/my_task/
├── card.yaml            # the task card (metadata, metric, file pointers)
├── initial_program.py   # the seed program, with an EVOLVE-BLOCK
├── evaluator.py         # evaluate(program_path) -> dict
├── Dockerfile           # the task's environment. This file IS the image — nothing is
│                        # inferred from the card, and a task without one cannot be
│                        # scored in a container at all.
└── README.md            # the human write-up (the card holds the LLM's prompt, not this)

The task owns the problem statement (task.context), the seed (task.initial_genome()), and the Evaluator (task.evaluator). Because Galapagos supplies the Evaluator from the task, runnable compatible scaffolds can reuse it unchanged.

Below is a complete, minimal task: tune three numbers to minimize their sum of squares.

1. `card.yaml`¶

name: my_task
display_name: My Task
domain: math
summary: "Tune a 3-vector to minimize the sum of squares."
system_message: |
  You are an expert in numerical optimization.
  Tune a length-3 PARAMS vector to MINIMIZE sum(x_i^2). Implement `solve() -> list` returning the
  three numbers. Only the code inside the EVOLVE-BLOCK is modified by the search. The score is
  1/(1+loss), recomputed independently from the returned values (anti reward-hacking).
metrics:
  - metric_name: combined_score
    metric_direction: maximize
    metric_description: "1 / (1 + sum of squares); higher is better."
    metric_computation: "Recompute sum(x_i^2) from the returned vector; score = 1/(1+loss)."
components: {initial_program: initial_program.py, evaluator: evaluator.py}
                                                 # no `environment:` block beyond the timeout below —
                                                 # it names what the EVALUATOR needs, not where the task
                                                 # runs (that is the run's general.eval_mode).
language: python
modality: text
environment: {est_runtime_s: 30}                  # sets the evaluator timeout (est*4+30)
references: {source: "my repo", best_known: 1.0, note: "score -> 1 as the vector -> 0"}

The system_message is what gets injected as task.context — the proposer's system prompt. Open it with a role line ("You are an expert …"), then the problem statement, the exact entry point, and the objective; leave provenance and infrastructure notes to the sibling README.md. components.initial_program and components.evaluator point at the two files. A runnable task must declare initial_program explicitly; there is no filename fallback or seed alias. Its suffix is checked against the canonical language (python → .py, cpp → .cpp) when the card loads. The metrics list declares the headline number and direction. Iterative search always calls evaluator.py::evaluate(program_path). If the same file also defines the optional evaluate_final(program_path) lifecycle hook, Galapagos automatically calls it exactly once against the frozen winner after search; no additional card component is needed.

Every task directory also needs a README.md; the submission gate checks for it. See the rendered READMEs under src/galapagos/tasks/*/README.md for the house template.

2. `initial_program.py`¶

The seed must mark the editable region with # EVOLVE-BLOCK-START … # EVOLVE-BLOCK-END. The Proposer only touches code inside those markers.

"""Tune the PARAMS vector to MINIMIZE the sphere loss sum(x_i^2) (score = 1/(1+loss))."""


def solve():
    # EVOLVE-BLOCK-START
    PARAMS = [0.9, -0.8, 0.7]
    # EVOLVE-BLOCK-END
    return PARAMS


if __name__ == "__main__":
    print(solve())

Keep the seed runnable and self-contained — it is evaluated once at setup to seed the population.

3. `evaluator.py`¶

The contract is one function: evaluate(program_path) -> dict returning at least combined_score (a float). By default Galapagos runs it inside your task's own container image (the ContainerEvaluator); it drops to an isolated host subprocess (the SubprocessEvaluator) when the run asks for it (general.eval_mode: local). Either way it must be importable and side-effect-free.

"""Deterministic evaluator. Contract: evaluate(program_path) -> dict with combined_score (maximize)."""
import importlib.util

N = 3


def _load(program_path):
    spec = importlib.util.spec_from_file_location("_cand", program_path)
    mod = importlib.util.module_from_spec(spec)
    spec.loader.exec_module(mod)
    return mod


def evaluate(program_path):
    try:
        params = list(_load(program_path).solve())
    except Exception as e:  # candidate crashed
        return {"combined_score": 0.0, "validity": 0.0, "status": "exec_error",
                "artifacts": {"text_feedback": f"execution error: {e}"}}
    if len(params) != N or any((not isinstance(x, (int, float)) or x != x) for x in params):
        return {"combined_score": 0.0, "validity": 0.0,
                "artifacts": {"text_feedback": f"PARAMS must be {N} finite numbers"}}
    loss = sum(float(v) ** 2 for v in params)
    score = 1.0 / (1.0 + loss)
    return {
        "combined_score": score,                 # the headline number (required)
        "loss": loss,                            # any extra numeric metrics are kept too
        "validity": 1.0,
        "artifacts": {"text_feedback": f"loss={loss:.4f}, score={score:.4f}"},
    }

The returned dict keys mean:

Key	Effect
`combined_score`	The headline fitness (required). Maximized by the loop.
any other numeric key	Kept in `genome.scores` and shown in the prompt's metrics section.
`validity` / `status`	Gate admission. `validity: 0.0` or `status` in `{exec_error, invalid}` marks the candidate invalid.
`artifacts.text_feedback`	Surfaced back into the next prompt under "Evaluator feedback".
`per_instance`	Optional per-test-case success vector (for Pareto / instance-level methods).

Recompute the score independently

Score the candidate from its returned output, not from any number it printed — this is the anti-reward-hacking discipline the bundled tasks follow. A candidate that returns combined_score = 999 from inside its own code gains nothing; the evaluator decides the score.

Optional: final evaluation after search¶

Use evaluate_final() when search should rank candidates on train data but the completed run should report one held-out score. Define it as a top-level function in the same evaluator.py:

def evaluate(program_path):
    train = load_train()
    return fit_and_score(program_path, fit_data=train, score_data=train)


def evaluate_final(program_path):
    train = load_train()
    held_out = load_held_out()
    return fit_and_score(program_path, fit_data=train, score_data=held_out)

Galapagos then applies this lifecycle automatically:

Every search iteration calls evaluate().
After search, the winner is frozen.
evaluate_final() runs once on that winner; a valid combined_score becomes score_of_record and is never returned to search.

Keep the card unchanged—only the main evaluator is a component:

components:
  initial_program: initial_program.py
  evaluator: evaluator.py

Do not add components.final_evaluator or a second evaluator file; that field is invalid. Ensure evaluate() never loads held-out data. This hook is for a bundled/public final protocol; secret or private leaderboard data belongs in a trusted verification service, not the task bundle.

Optional: staged (cascade) evaluation¶

Besides the plain evaluate, an evaluator may define evaluate_stage1 (and optionally evaluate_stage2 / evaluate_stage3). When the stages exist, Galapagos runs them in order in the same subprocess, aborting early when a stage's combined_score (or, if it returns none, the average of its numeric metrics) falls below its gate — so a cheap smoke test can gate an expensive full evaluation. Both knobs live in the card's environment block: cascade_evaluation: true and cascade_thresholds: [0.3, 0.6] (the defaults). A task with only a plain evaluate is unaffected.

Optional: LLM-backed judging¶

An evaluator that calls an LLM can opt into the run's typed EvaluatorConfig by declaring the exact keyword-only parameter evaluator_config. Galapagos passes it as a plain dict in local and Docker mode; ordinary one-argument evaluators remain unchanged.

import os


def evaluate(program_path, *, evaluator_config):
    candidate = open(program_path).read()
    api_key = os.environ[evaluator_config["api_key_env"]]
    verdict = call_judge_llm(                 # implemented by this task
        candidate,
        model=evaluator_config["model_path"] or evaluator_config["model_name"],
        api_base=evaluator_config["api_base"],
        api_key=api_key,
        temperature=evaluator_config["temperature"],
        max_tokens=evaluator_config["max_tokens"],
    )
    return {"combined_score": float(verdict.score),
            "artifacts": {"text_feedback": verdict.feedback}}

Place the call settings in the run config, while keeping the rubric and response parser in the task's evaluator.py:

evaluator:
  model_name: openai/gpt-5-mini
  api_base: openrouter
  api_key_env: OPENROUTER_API_KEY   # a variable name, never the secret value
  temperature: 0.0
  max_tokens: 2048

The same opt-in parameter works on evaluate_stage1/2/3 and evaluate_final. LLM verdicts are inherently less reproducible than executable checks; the task-eligibility policy therefore does not allow one as the sole leaderboard score, even though the runtime supports LLM-assisted evaluators.

Validate it¶

gx.available_tasks() and galapagos task list show the cards bundled in the galapagos package (under src/galapagos/tasks/); a third-party task does not appear there and does not load by name — load it by explicit path to its card:

import galapagos as gx
task = gx.GalapagosTask.from_card(path="tasks/my_task/card.yaml")
task.runnable                       # -> True  (seed + evaluator present)
task.context                        # the problem text
seed = task.initial_genome()
print(task.evaluator.evaluate(seed).combined_score)  # score the seed directly

Run a compatible scaffold against it:

scaffold = gx.OpenEvolveScaffold.from_card(model=gx.load_model("openai/gpt-4o-mini", host="openrouter"))
result = scaffold.run(task=task, max_iterations=30)
print(result.best_score)

Docker (sandbox) evaluation¶

By default your task is scored in a container — task.evaluator is a ContainerEvaluator that runs your evaluator.py inside an image the task itself declares (Harbor-style: the task owns its environment), so a deterministic scorer runs the same way on every machine. That image is the Dockerfile you ship — there is no second source, and nothing is inferred from the card.

environment:
  # the image and the cascade knobs. Where the task is SCORED is not a card setting — that is the
  # run's `general.eval_mode`. All optional, all with sensible defaults:
  # dockerfile: Dockerfile # the task's Dockerfile — the ONE image key, and only needed if it is not
                           # simply named `Dockerfile`. Everything else an image needs, the Dockerfile
                           # says: a prebuilt tag is `FROM tag`, pinned deps are `RUN pip install`.
  # env: [HF_TOKEN]        # host env vars to forward in (a list of names), or a literal {KEY: value} map

Ship a Dockerfile. It is usually two lines:

FROM galapagos-task-base:0.4.0            # or -all / -cuda / -algotune
RUN pip install --no-cache-dir numpy

It must start FROM a galapagos base. FROM python:3.12-slim is the natural, wrong answer: the base image has open-galapagos preinstalled, and several evaluators import wheel-only sibling modules (e.g. galapagos.tasks.kernelbench_common) that a bare python image simply does not have — the scorer would not import, and every candidate would score a silent zero. The submission gate refuses a Dockerfile that gets this wrong, and refuses a task that ships none.

It builds with the task folder as its context, so it may COPY task files — but it rarely needs to: your evaluator.py and its data are copied into the running container anyway, so the Dockerfile only has to describe the environment, not the task. How the base-image family, content-addressed image tags, the one-long-lived-container-per-run lifetime, and GPU wiring work is documented under Task environments.

Your card does not choose where it runs — the operator does, with general.eval_mode (galapagos run --general.eval_mode local). It is the only input, so a run is reproducible from its config alone. If it is your machine that has no Docker, that is a fact about your machine, not your task: pass --general.eval_mode local and leave the card alone, so the task stays correct for everyone else.

If your task genuinely cannot be containerized at all (the only bundled example is the Apple-Metal mlx_metal_kernel_opt task), declare the submission exception described above, explain it under environment.requires, and run it with --general.eval_mode local — you lose the reproducibility above, which is why it is a last resort. The same run-level override is available from the Python API by calling set_eval_mode directly on the task object before running it:

task = gx.GalapagosTask.from_card(path="tasks/my_task/card.yaml").set_eval_mode("docker")
print(task.evaluator.evaluate(task.initial_genome()).combined_score)  # scored in the sandbox

The sandbox is sealed: unlike the local subprocess (which inherits the host environment, minus the provider LLM keys — those are scrubbed unless explicitly authorized), a docker-mode evaluator sees only the env vars you list under environment.env, the keys declared under environment.api_keys, and the one key named by the run's evaluator.api_key_env. The latter is the normal run-level path for an LLM judge; the secret value itself never enters the serialized config. Docker mode needs the docker CLI on PATH; if it's missing, the evaluator raises with a clear message. For scoring that reproduces across machines (not just across the local/container split on one machine), ship a Dockerfile with a pinned base — and it travels: galapagos submit uploads it, so the Hub copy of your task is scored in the image you wrote.

To publish it, see Submit to the Hub.

Write your own task¶

1. card.yaml¶

2. initial_program.py¶

3. evaluator.py¶