Contributing¶

Galapagos accepts four main contribution paths: task submissions, scaffold submissions, executable upstream reviews of task ports, and executable upstream reviews of scaffold ports. Core library and documentation pull requests are also welcome.

This guide takes a first-time contributor from a fork to a passing pull request. A task or scaffold is not considered portable merely because it works inside the source checkout: its clean galapagos submit bundle must also load and run.

Contribution flow¶

1. Fork and clone¶

Fork Open-Galapagos/galapagos-dev on GitHub, then clone your fork and keep the canonical repository as upstream:

git clone https://github.com/<YOUR_GITHUB_USERNAME>/galapagos-dev.git
cd galapagos-dev
git remote add upstream https://github.com/Open-Galapagos/galapagos-dev.git
git fetch upstream

2. Create one branch per contribution¶

Start from the latest upstream main:

git switch -c contrib/<short-description> upstream/main

If the branch already exists, update it before continuing:

git fetch upstream
git rebase upstream/main

Do not mix unrelated task, scaffold, formatting, or generated-file changes in the same pull request.

3. Set up the development environment¶

python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -e '.[math,dev]'

On Windows, activate the environment with .venv\Scripts\activate.

4. Pick the matching path¶

Contribution	Directory	Local command	PR template	GitHub check
Task submission	`src/galapagos/tasks/<slug>/`	`python scripts/validate_task_submission.py src/galapagos/tasks/<slug>`	`task_submission.md`	`submit-task`
Scaffold submission	`src/galapagos/scaffolds/<slug>/`	`python scripts/validate_scaffold_submission.py src/galapagos/scaffolds/<slug>`	`scaffold_submission.md`	`submit-scaffold`
Upstream task-port review	`porting_reviews/tasks/<slug>/`	`python scripts/validate_task_port_review.py porting_reviews/tasks/<slug>`	`task_port_review.md`	`validate-task`
Upstream scaffold-port review	`porting_reviews/scaffolds/<slug>/`	`python scripts/validate_scaffold_port_review.py porting_reviews/scaffolds/<slug>`	`scaffold_port_review.md`	`validate-scaffold`

submit-* checks portability of a normal contribution. validate-* runs a separate fidelity test authored by an upstream author or maintainer. The four workflows are path-scoped, so a focused pull request triggers only its matching check.

Contribute a task¶

Research eligibility gate¶

A task is eligible for the research catalog and leaderboard only when it satisfies all three of the task eligibility requirements:

Unsolved: no known reproducible solution has achieved the documented perfect score under the submitted protocol and resource budget.
Open-ended: it is a genuine research or optimization problem with multiple improvement paths and continuous or finely graded scoring—not merely a pass/fail exercise.
Deterministically verifiable: a trusted evaluator independently recomputes the score from candidate output using pinned fixtures, seeds, dependencies, budgets, and tolerances.

The pull request must define perfection, cite the best known result and remaining gap, describe the available optimization axes, and explain the evaluator's reproducibility boundary. Automated checks verify packaging and execution; maintainers review these research claims.

Required layout¶

Create one self-contained directory:

src/galapagos/tasks/<slug>/
├── card.yaml
├── README.md
├── initial_program.py
├── evaluator.py
├── Dockerfile
└── ... support files declared by the card

The directory name and card.name must match. A minimal task card describes the prompt, components, language, and headline metric:

name: my_task
display_name: My Task
domain: optimization
summary: A one-line description of the problem.
system_message: |
  Implement solve() and return a candidate solution.
components:
  initial_program: initial_program.py
  evaluator: evaluator.py
language: python
metrics:
  - metric_name: combined_score
    metric_direction: maximize

See Write your own task for the complete schema and evaluator contract.

language is a canonical task-card value, not a filename shorthand: use python, not py, and cpp, not c++. Its declared initial_program must use the matching extension (.py and .cpp respectively); the card loader rejects a mismatch before a run begins.

Evaluator requirements¶

The evaluator contract is evaluate(program_path) -> dict. It must:

recompute validity and scores from candidate output;
return a finite combined_score;
reject malformed or known-invalid output;
be deterministic for the same candidate and fixtures;
avoid secrets, paid services, and hidden local state.

Never accept a score reported by the candidate itself. Evaluator-owned fixtures and support files must be part of the uploaded bundle.

Environment and assets¶

Every ordinary task ships a Dockerfile, and every stage derives from a Galapagos task base:

FROM galapagos-task-base:0.4.0 AS builder
# build dependencies

FROM galapagos-task-base:0.4.0
# runtime dependencies

Declare small supporting files in the card's assets. Use external resources for large data. A task must not pass only because an undeclared sibling file is available in the source checkout.

The default seed smoke test runs locally with a CI time cap. An exceptional task may make a different policy review-visible:

metadata:
  submission:
    smoke_test: docker       # local (default) | docker | skip
    smoke_timeout_s: 120     # capped at 300 seconds
    reason: "Why the non-default policy is necessary."

skip requires a reason. A task that genuinely cannot run in Linux may also set host_only: true with that reason.

Validate before opening the PR¶

python scripts/validate_task_submission.py src/galapagos/tasks/<slug>

This command:

runs the real galapagos submit --repo-type task --dry-run path;
materializes only the files that would be uploaded;
reloads that clean bundle with load_task;
checks that every file named under components exists in the clean bundle;
checks the card, prompt, README, seed, evaluator, assets, and Dockerfile;
sends the seed through the evaluator and requires a finite verdict.

The seed need not already solve the problem. Some discovery tasks deliberately start outside the feasible region. The gate requires a working evaluator, not a completed discovery.

Contribute a scaffold¶

Required layout¶

src/galapagos/scaffolds/<slug>/
├── card.yaml
├── README.md
├── scaffold.py
└── ... support files declared by the card

See Build your own scaffold for the complete implementation walkthrough.

A scaffold maps a method onto Galapagos's population, selection policy, prompt builder, proposer, task-owned evaluator, and memory interfaces. A runnable card must either:

name an importable GalapagosScaffold subclass in controller; or
provide executable component paths that resolve from the upload bundle.

A design-only card must explicitly use status: spec. Evaluation remains task-owned, so the scaffold card uses components.evaluator: {kind: task}.

Validate the exact artifact:

python scripts/validate_scaffold_submission.py src/galapagos/scaffolds/<slug>

The command performs the same submit/bundle round trip and requires all six component slots. It then imports the controller or resolves executable components from the clean bundle. For a controller-backed scaffold it calls build_components() and verifies that the five scaffold-owned slots produce the correct Galapagos component types; the evaluator remains task-owned. Reusing a component from Galapagos core or another scaffold is valid. The command never calls a live or paid model API.

Review somebody else's port as an upstream author¶

Mechanical CI cannot determine whether a port faithfully represents a paper or codebase. If a contributor ported work they did not author, the normal submission PR should identify the upstream source and invite an upstream first author, corresponding author, original author, or repository maintainer to open a separate executable review.

The port should normally be on main before the review PR is opened. If the reviewer finds a problem, the review PR may include the required corrections.

Task port¶

mkdir -p porting_reviews/tasks
cp -r porting_reviews/templates/task porting_reviews/tasks/<slug>
# edit review.yaml and replace the assertions in test_port.py
python scripts/validate_task_port_review.py porting_reviews/tasks/<slug>

Scaffold port¶

mkdir -p porting_reviews/scaffolds
cp -r porting_reviews/templates/scaffold porting_reviews/scaffolds/<slug>
# edit review.yaml and replace the assertions in test_port.py
python scripts/validate_scaffold_port_review.py porting_reviews/scaffolds/<slug>

The review must come from the GitHub account recorded in reviewer.github. Maintainers separately verify public evidence connecting that account to the paper or canonical repository. Review tests must be deterministic, offline, secret-free, and specific to the upstream contract.

CI runs test_port.py against a temporary directory containing only the Hub bundle. After merge, later changes to the reviewed task or scaffold rerun the same upstream-authored test. See Review a port as an upstream author.

Contribute core code or documentation¶

Keep a core change focused and add a regression test for behavior changes. Evaluation belongs to the task, not the scaffold. Avoid importing heavy optional dependencies on the core search path.

Run the smallest relevant test first, then the broader suite where practical:

pytest tests/<relevant-test-file>.py
pytest

For documentation changes, install the documentation tool and build the site locally:

python -m pip install mkdocs-material
mkdocs build --strict -f docs/mkdocs.yml

Open the pull request¶

Review and stage only the files for this contribution:

git status --short
git diff --check
git add <paths-for-this-contribution>
git commit -m "Describe the contribution"
git push -u origin contrib/<short-description>

Open a pull request from your fork to Open-Galapagos/galapagos-dev:main. Choose the matching template under .github/PULL_REQUEST_TEMPLATE/ and include:

a concise explanation of what changed and why;
canonical paper, repository, data, and license links where applicable;
the successful local validation summary;
intentional differences from upstream;
focused test results for core changes.

Never commit credentials, API keys, private data, local virtual environments, or generated build output. Push fixes to the same branch while the pull request is open; GitHub updates the PR and reruns the checks automatically.

Models and discoveries¶

Model configuration is local and does not use this executable submission gate. Discovery results use the Hub verification flow described in Submit to the Hub.