Installation¶

Install

pip install "open-galapagos[all]" — published as open-galapagos (the bare galapagos name is taken on PyPI), but you still import galapagos.

Python

≥ 3.10

Docker

The default evaluation path — a task is evaluated inside the image it declares. Use --general.eval_mode local for a host subprocess.

Then

For API-model scaffolds, set a provider key — Configure a provider. Claude-backed local modes use subscription auth instead.

A task is evaluated inside the container image it declares — the task owns its environment, which is what makes a score reproducible across machines. On a host with no Docker, evaluate in a host subprocess instead: galapagos run --general.eval_mode local (a config value like any other — you can also put general: {eval_mode: local} in a --config YAML). Both modes run the same evaluator source, but dependencies, hardware, isolation, and timing can differ; local mode needs the task's dependencies on the host rather than supplied by its image.

From PyPI¶

pip install open-galapagos                 # core only
pip install "open-galapagos[all]"          # core + every general-purpose extra

With uv: uv pip install "open-galapagos[all]".

This pulls the lightweight core (the frozen-search path): an OpenAI-compatible client, pydantic, pyyaml, rich, and python-dotenv. After install, import galapagos as gx is ready to use and the galapagos console script is on your PATH.

From source¶

Install an editable checkout from the repository:

git clone https://github.com/Open-Galapagos/galapagos.git
cd galapagos
pip install -e .                   # core only
pip install -e ".[all]"            # core + every general-purpose extra

Optional extras¶

The core is deliberately minimal. Heavier capabilities live behind extras, so a plain install stays light and fast:

Extra	Pulls in	Enables
`math`	`numpy`, `scipy`, `shapely`, `sympy`, `jax`, `optax`, `tqdm`	the 17 SkyDiscover math tasks and other numeric tasks
`algotune`	`cvxpy`, `ortools`, `scikit-learn`, `pot`, `faiss-cpu`, `python-sat`, `highspy`, …	AlgoTune evaluators in local mode; the benchmark-faithful candidate toolbox is in `task-base-algotune`
`frontiercs`	`requests>=2.32.5`	the FrontierCS 1.0 algorithmic suite (188 tasks) — talks to the pinned go-judge
`dev`	`pytest`	running the test suite
`all`	`math`, `algotune`, `frontiercs`	mutually compatible host extras; GPU/ALE/FrontierCS 2.0/SLD-Bench environments remain task-owned

pip install "open-galapagos[math]"        # numpy / scipy for numeric tasks
pip install "open-galapagos[algotune]"    # AlgoTune's host-side evaluator solver stack
pip install "open-galapagos[frontiercs]"  # requests, for the FrontierCS C++ judge client
pip install "open-galapagos[all]"         # every general-purpose extra at once
pip install "open-galapagos[dev]"         # pytest, for running the test suite

Some bundled tasks need extra libraries

Galapagos ships a large catalogue of benchmark tasks, and a task's evaluator runs real code. Most numeric tasks are covered by [math]. In Docker mode the full AlgoTune suite (154 tasks — convex/discrete optimization, graph, ODE/PDE, crypto, …) uses its dedicated source-faithful image, including upstream's NumPy 1.26.4 and CPU Torch 2.7.0 pins. A few other specialized suites declare their own requirements — e.g. some cloud or Apple-silicon kernel tasks need modal or mlx. A task's Dockerfile lists exactly what it needs; install those on demand, or start with the numeric tasks, which run on core + [math] alone.

What the bundled suites need¶

The 1,249 bundled task cards span several benchmark suites; 1,244 have a runnable seed/evaluator pair. Most run on core + [math]; the ones that need more are:

Suite	Tasks	To run it, you need
SkyDiscover math	17	Docker: `task-base` supplies `[math]`, including JAX/Optax and SymPy. Local: install `[math]`
AlgoTune	154	Docker: `task-base-algotune`, with upstream NumPy/scientific pins, CPU `torch==2.7.0`, and the full optimization toolbox. Local `[algotune]` is evaluator-compatible but not the benchmark timing environment
FrontierCS 1.0	188	Docker task image supplies `requests>=2.32.5`; start the pinned external go-judge — Docker and a C++17 toolchain (see below)
FrontierCS 2.0	14	two Erdos tasks need only `task-base`; standard/relaxed vector ANN images bundle Rust + `faiss-cpu==1.11.0`; disk ANN still needs SIFT100M data; the remaining BBOPlace/DuckDB/Generals/NanoWM/vLLM tasks require the external runtime/data named by their cards
LLM-SRBench (symbolic regression)	240	`numpy`, `scipy`, `datasets`, `huggingface_hub`, `h5py`, plus one-time access approval and a cached copy of the gated `nnheui/llm-srbench` dataset (see below)
SLD-Bench (scaling-law discovery)	8	dedicated Python 3.13 Docker base pins `datasets==4.0.0`, `huggingface-hub==0.34.4`, `numpy==2.3.2`, `scipy==1.16.1`. Dataset revisions are pinned in the evaluators; see the separate local-mode setup below
AlphaEvolve (math discovery)	60	core + `[math]` (a few also use `sympy` / `networkx` / `shapely`)
GPU MODE (kernels)	46	the four SkyDiscover tasks audited here use CUDA: grayscale/trimul/vecadd pin `torch==2.7.1+cu128` + `triton==3.3.1`; MLA decode overlays `torch==2.8.0+cu128` + `triton==3.4.0`. Other GPU tasks declare their own hardware/runtime
KernelBench	270	an NVIDIA CUDA GPU plus `torch` / `triton`. In the default docker mode a GPU-less host fails loudly: the card declares `gpu`, so the container asks for a device and Docker's "could not select device driver" becomes an actionable error. It is only on the opt-in `--general.eval_mode local` path that the harness returns `combined_score` 0.0 / `validity` 0.0
ALE-Bench (AtCoder heuristic)	40	Docker, a uid that can reach the daemon socket, and the pre-built judge images — the evaluator drives Docker itself. Nothing to pip-install: `ale_bench` ships inside the task image (see below)
Open Problems (single-cell)	16	the single-cell stack — `anndata`, `scanpy`, `scikit-learn`, …

LLM-SRBench gated dataset setup¶

The 240 lsr_* tasks use nnheui/llm-srbench. Its metadata page is public, but its files are gated: the Hugging Face account downloading them must first accept the dataset's access conditions. Prepare each account and machine once before starting an agent run:

Open the dataset page, sign in, and accept or request access as prompted.
Authenticate the machine:
```
hf auth login
hf auth whoami
```

Download the exact revision used by the bundled evaluators:

hf download nnheui/llm-srbench \
  --type dataset \
  --revision 4f22d48100d94125d3af1e7bd40897fcc7f9597d

Do not add --local-dir: Galapagos expects a Hub cache. The default is ~/.cache/huggingface; HF_HOME selects another location. In Docker mode Galapagos mounts only the datasets--nnheui--llm-srbench repository cache, read-only, for both the coding agent and evaluator. It does not mount your Hugging Face home or login token, and Hub access inside the container is forced offline. The bundled Codex LLM-SRBench sweep checks the pinned snapshot before starting Codex. A 401 Unauthorized error means the active account has not been granted access, or the selected HF_HOME does not contain the downloaded snapshot.

Requirements outside these families still vary by card. The quickstart tasks circle_packing and function_minimization run with core + [math]; for any other task, treat its Dockerfile, environment metadata, and README as the source of truth.

Running the FrontierCS algorithmic suite (Frontier-CS/problem_<id>)

The 188 FrontierCS algorithmic tasks (ICML'26, arXiv:2512.15699) take a single-file C++17 solution and score it with FrontierCS's own go-judge (a Node + go-judge sandbox that compiles, runs every test case under the time/memory limit, and applies the problem's checker/interactor). The galapagos evaluator just submits your solution to that judge over HTTP, so to actually run these tasks you need:

Docker + a C++17 toolchain (g++) — to build and run the judge.
pip install "open-galapagos[frontiercs]" — installs requests>=2.32.5 for local mode. Docker mode already installs that thin layer in each task image.
Install the audited FrontierCS checkout and start the judge once (first run builds the go-judge image, mounts the official problems directory, and listens on :8081):
```
bash scripts/setup_frontiercs1_judge.sh
```

The script checks out commit 6d597dfb60be9e592881aef051b94e30d197c436, the source of truth used to adapt the 188 tasks. Set FRONTIERCS_ROOT to choose another checkout location.

Point the evaluator elsewhere with FRONTIERCS_JUDGE_URL (default http://localhost:8081). Without a reachable judge, a FrontierCS task's evaluate() returns a structured zero-score explaining the missing dependency — the cards still load and list, they just can't be scored until the judge is up. The score is continuous (0–100; the unbounded score can exceed 100 when a solution beats the reference), so there is always room to improve.

Running the ALE-Bench AtCoder heuristic suite

The 40 ALE-Bench tasks (SakanaAI) — 38 named ahc*, plus future_contest_2022_qual and toyota2023summer_final — are AtCoder Heuristic Contest problems: a single-file C++20 program, scored by the official judge on the contest's 50 public cases. They are the one suite where the evaluator is itself a Docker client — ale_bench compiles and runs each candidate inside sibling judge containers it launches on your daemon. It needs the daemon, not merely a container, and it exposes no judge service to call instead (unlike FrontierCS's go-judge above, which is an HTTP server). Setup is therefore about Docker, not pip:

A running Docker daemon your uid can reach. The socket is root:docker, so join the group once (sudo usermod -aG docker $USER, then re-login). Galapagos preflights this before it builds anything and fails loudly with the fix — it never silently scores zero.
Build the judge images once (~25 GB for all eight language/version tags; the current problems use cpp20-202301, 3.9 GB):
```
bash scripts/setup_ale_bench_judge_images.sh
```
This pulls upstream's prebuilt images and derives local ale-bench:<lang>-<version> tags with a world-writable /workdir, so the judge compiles whatever uid you run as. Skip this and every candidate scores the error floor: ale_bench looks the images up by bare name, fails to pull a repo that does not exist, and gives up.
Nothing to pip-install. ale_bench is not on PyPI; the task's own image installs it from source. (You only install it yourself for --general.eval_mode local — see the caveats below.)
Network on the first run. The first session fetches the problem's assets from the Hugging Face Hub and builds its Rust judge tools (inside a container — no host Rust toolchain needed). Both are cached in a persistent parity directory, ~/.cache/galapagos/dood (override with $GALAPAGOS_DOOD_DIR), so later runs skip it.

Then run them like any other task — the whole loop, or just the evaluator, moves into the container for you:

galapagos run --scaffold openevolve --task ahc001 \
  --proposer.model_name openai/gpt-5.5 --proposer.api_base openrouter --general.max_iterations 50

# the Claude Code agent, inside the task's own image
galapagos run --scaffold claude_code --task ahc001 \
  --general.max_iterations 50 --proposer.claude_wall_timeout_seconds 7200

Four things to know before you trust a number:

The container is not a sandbox for this suite. Granting the daemon socket is granting root on the host. For an ordinary scaffold that is harmless — the trusted evaluator holds the socket and the candidate never sees it. But claude_code puts an agent with a shell in that container, so on an ahc* task the agent can reach your daemon, and therefore your host. Run it only on a machine you are willing to expose. (The alternative is not safer: a host-side agent runs as you, and you must be in the docker group for any of this to work at all.)
Set proposer.claude_wall_timeout_seconds explicitly. These cards declare a long est_runtime_s, so the evaluator timeout is ~9,600 s and claude_code's derived session timeout (cli_max_turns × (300 + eval_timeout)) becomes days. The safety net will not save you unless you set it.
Scoring is slow, and it is the real judge. Each candidate compiles and runs 50 cases under the contest's per-case time limit — tens of seconds to minutes. combined_score is the official absolute score × optim_factor / 50 (optim_factor is −1 for minimize-type problems, so negative scores are legitimate there). An evaluator failure floors at -1e300 with validity: 0, so a crash can never outrank a real negative score — -1e9 was not low enough, since ale_bench's own reject value is -sys.maxsize-1.
You want ≥ 13 physical cores. The evaluator asks ale_bench for 13 parallel judge workers, each pinned to 1 CPU. On a smaller host they contend, the judge's wall-clock measurement inflates, and a good solution can be rejected with a spurious TLE.

Prefer not to hand a container the socket? Score on the host instead with galapagos run --general.eval_mode local. Then you provide what the image otherwise would: sudo apt install libcairo2-dev libffi-dev and pip install "ale_bench[eval] @ git+https://github.com/SakanaAI/ALE-Bench.git@8ae5b832227e50729f7673f248565c319c301264" (Python 3.10–3.14). The same audited commit is pinned in all 40 task images. Without them the evaluator returns its structured -1e300 / validity: 0 failure naming the missing package rather than a misleading zero.

Running the four audited GPU MODE tasks

Docker is the reproducible path. The dedicated task-base-gpu-mode image is nvidia/cuda:13.0.1-devel-ubuntu24.04 plus PyTorch's cu128 torch==2.7.1 wheel, whose exact dependency is triton==3.3.1. That matches grayscale, trimul, and vecadd. mla_decode has a task layer that installs torch==2.8.0 from the same cu128 index, which pins triton==3.4.0. All four cards request a GPU, so Docker also needs the NVIDIA Container Toolkit on the host.

For local mode, keep the two environments separate; installing the MLA pair replaces the other three tasks' pair:

# grayscale / trimul / vecadd
pip install --index-url https://download.pytorch.org/whl/cu128 "torch==2.7.1"

# mla_decode, in a separate virtual environment
pip install --index-url https://download.pytorch.org/whl/cu128 "torch==2.8.0"

Running SLD-Bench locally

Docker mode needs no host scientific packages: all eight tasks inherit task-base-sldbench, which matches upstream's Python ≥3.13 requirement and exact dependency versions. Local mode must use a separate Python 3.13 environment:

python3.13 -m venv .venv-sldbench
. .venv-sldbench/bin/activate
pip install open-galapagos \
  "datasets==4.0.0" "huggingface-hub==0.34.4" \
  "numpy==2.3.2" "scipy==1.16.1"
galapagos run --task sldbench_vocab --general.eval_mode local

These pins intentionally are not part of [all]: an optional dependency requiring Python 3.13 would make the otherwise Python 3.10-compatible Galapagos package impossible to resolve across its declared support range. The task-owned Docker base keeps both contracts honest.

external/ — where task-specific runtime deps live (kept local)

Some suites need a service or repo that isn't a pip package — the FrontierCS go-judge above is the first example. By convention these live under external/ at the repo root: a gitignored, local-only area for task-specific runtime dependencies you clone or install on demand (judge servers, sandboxes, large external repos). It is never committed and is not part of the installed package, so each machine sets up only the suites it actually runs. A task's evaluator reaches its service over a configurable endpoint (e.g. FRONTIERCS_JUDGE_URL), so the exact clone location doesn't matter to scoring — external/ is simply the tidy, ignored home for it.

Running the CLI-agent scaffolds

The claude_code scaffold drives a local Claude Code CLI session, billed through your Claude subscription (no API key). It needs the claude binary on your PATH (npm install -g @anthropic-ai/claude-code) and a login — run claude and /login once, or set CLAUDE_CODE_OAUTH_TOKEN from claude setup-token for headless runs. For a docker-mode task (the default) claude_code runs the session inside the task's own container, which needs Docker on the host (it builds the task's image and layers the claude CLI onto it on first run — the host claude binary is only needed for local-mode tasks). See Task environments for how a task resolves to an image.

meta_harness also defaults to Claude Code, but it requires --output-dir; its archive is written to <output-dir>/meta_harness_D. In Docker evaluation mode the whole Meta-Harness loop runs inside a Claude-enabled derivative of the task image. It does not support --resume.

The codex scaffold runs one codex exec --json session. Docker-mode tasks install @openai/codex into the task image automatically; local mode requires the codex binary on the host. Authentication is ChatGPT-managed rather than API-key billed: set CODEX_ACCESS_TOKEN when your workspace provides one, or set cli_auth_credentials_store = "file" in ~/.codex/config.toml and run codex login. The scaffold isolates that login in its run directory, strips API-key provider variables, and safely writes refreshed credentials back after the run.

Configure a provider¶

Galapagos talks to any OpenAI-compatible endpoint. The host= you pass to GalapagosModel.from_card(...) selects the client, and each host reads its own key and endpoint env vars (OPENAI_BASE_URL is consulted only for host=openai — every other host has its base URL hardcoded or reads VLLM_BASE_URL / LITELLM_BASE_URL / AZURE_OPENAI_BASE_URL). See the host table in Models. The galapagos CLI reads a .env file automatically (searched upward from the current directory; real environment variables win). When using the Python API directly, load it yourself first (from dotenv import load_dotenv; load_dotenv()) or export the variables in your shell:

OpenRouterOpenAILocal vLLM / Ollama

# .env
OPENAI_API_KEY=sk-or-...
OPENAI_BASE_URL=https://openrouter.ai/api/v1

# .env
OPENAI_API_KEY=sk-...
# (leave OPENAI_BASE_URL unset)

# .env
OPENAI_API_KEY=EMPTY
OPENAI_BASE_URL=http://localhost:8000/v1

The supported hosts are openai, openrouter (the default), anthropic, gemini, azure, vllm, and litellm — chosen with one host= argument; any other value containing :// is treated as an explicit OpenAI-compatible base URL. See Models for the full host table and per-host credentials.

Most runs need an API key

Every model-driven scaffold calls a live LLM, so you need a provider key before the first run. The reference setup is an OpenRouter key in OPENAI_API_KEY (e.g. OPENAI_API_KEY=sk-or-..., with OPENAI_BASE_URL=https://openrouter.ai/api/v1), set in the environment or a .env file as shown above. The exception is the claude_code scaffold, which bills through your Claude subscription instead — it needs a logged-in Claude Code CLI (claude + /login, or CLAUDE_CODE_OAUTH_TOKEN from claude setup-token), not a provider key.

Verify¶

import galapagos as gx

print(gx.available_scaffolds())     # ['adaevolve', 'ale_agent', 'algotune_agent', 'beam_search', 'best_of_n', 'best_of_n_attempts', 'claude_code', 'codex', 'evox', 'meta_harness', 'openevolve', 'topk']
print(gx.registered_scaffolds())    # the runnable subset — currently the same twelve
print(gx.available_tasks())         # every bundled task card

Or from the command line:

galapagos --version
galapagos scaffold list
galapagos task list

Next: the Quickstart.