Releasing peclet — the workflow¶

The repeatable procedure for a family release of the suite: every PyPI package (CPU and CUDA), the HPC containers, the Snellius and LUMI site packages, the GitHub Releases + Zenodo DOIs, the documentation sites, and the example gallery. It is written so that the next release can be executed by following it top to bottom. The one-off state for the release currently being prepared lives in RELEASE_PREP.md; this file is the durable process.

Releases so far: 0.1.0 / 0.2.0 (2026-07-02), 0.3.0 … 0.6.0 (2026-07-03 … 07-25), 1.0.0 (2026-09-12) … 1.4.0 (2026-10-04). The 1.4.0 family is what PyPI serves today (see §4 for the version table). Everything below was reconstructed from those releases, the workflows in each repo, and the traps they hit.

0. What a release consists of¶

Artifact	Where	Produced by	Status of the mechanism
CPU wheels + sdists: `peclet-core` (sdist only), `peclet-amr` (sdist only), `peclet-morton`, `peclet-flow`, `peclet-pnm`, `peclet-dem`, `peclet-voro`, `peclet-coupling` (sdist only)	PyPI	each submodule's `.github/workflows/release.yml` on a `v*` tag (cibuildwheel, manylinux x86-64 + aarch64, win_amd64, macOS arm64, cp310–cp314, `PECLET_VENDOR_DEPS=ON`; Trusted Publishing)	works (used for every release)
`peclet` metapackage (== pins on the CPU family; extras `[mpi]`, `[cfd-dem]`)	PyPI	umbrella `.github/workflows/release.yml` on a `v*` tag	works
CUDA wheels `peclet-flow-cu13`, `peclet-pnm-cu13`, `peclet-dem-cu13`, `peclet-voro-cu13`	PyPI	`cuda-wheel` job in the same `release.yml` (manylinux container + CUDA 13 toolkit, Kokkos-CUDA static, `nvidia-cuda-runtime` dependency wheel, rpath; one job per interpreter, cp310–cp314 — the same list as the CPU wheels, RELEASE_PREP 1.0–1.1 §11.6)	works; all four published at 1.0.1
Containers `peclet-cpu`, `peclet-cuda:-sm80/-sm90`, `peclet-hip:-gfx90a`	GHCR (`oras://ghcr.io/computational-chemical-engineering/...`)	umbrella `.github/workflows/containers.yml` on a `v*` tag or dispatch	cpu + cuda publish; hip builds since 2026-09-04 (failed on every tag 0.3.0–0.6.0, §8)
Snellius package	`/projects/0/prjs1022/peclet` (install script + wheelhouse) and the GHCR CUDA image	`tools/hpc/install_snellius.sh` run from the release tag (§7)	script validated for flow; family-wide script is new
LUMI package	HIP container + `tools/hpc/install_lumi.sh`	§8	image builds (since 2026-09-04); untested on AMD hardware
GitHub Releases (one per repo + umbrella) → Zenodo version DOIs	GitHub / Zenodo	`gh release create` (Zenodo webhook mints the DOI)	works
Docs: umbrella MkDocs site, per-repo Doxygen, Python API pages	GitHub Pages	`site.yml` (umbrella, on push to main touching docs/), `docs.yml` per repo; `tools/gen_python_api.py` generates `docs/python/*.md` at build time, inside the released `peclet-cpu` image	works; `containers.yml` re-runs the site after a release (§5.1); the site pages carry no version literals (§5.2)
Example gallery	`peclet-examples` Pages	authors re-execute and commit `_freeze/`; CI renders only	works; re-execution is manual and slow (§10)
CHANGELOG.md	umbrella	by hand	stale since 0.2.0

Dependency order matters because wheels vendor sibling headers at a pinned tag (cmake/PecletDeps.cmake: PECLET_TPX_TAG = core tag, PECLET_MORTON_TAG), and the metapackage pins with ==:

core, morton  →  flow, pnm, dem, voro  →  coupling  →  peclet (metapackage)  →  containers, sites, gallery

1. Choosing the moment (freeze criteria)¶

Do not start the workflow until all of the following hold. They are cheap to check and each one has cost a day when skipped.

No unmerged worktrees. Every flow-*, tel/* and other worktree branch is either merged to the submodule's main or explicitly parked (its work is not in the release). List them with git -C worktree list and git -C branch --no-merged main.
Every checkout is at origin/main. tools/release/check_release_state.sh reports behind/ahead per submodule; all must be 0. (The flow/ checkout was 55 commits behind origin/main on 2026-09-04 because the VoF campaign landed through worktrees — a release cut from that checkout would have missed the whole campaign.)
Umbrella pointers == submodule HEADs and the umbrella git status is clean apart from the known scratch directories.
CI green on main of every repo (gh run list -R computational-chemical-engineering/<repo> -b main -L 3).
The local full test matrix (§3) passes on host and CUDA, including the MPI suites.
No open "silently wrong" issues in docs/SCALING_ISSUES.md or peclet-examples/ISSUES.md that a user would hit through the public Python API in a default configuration. Known limitations are fine if they are listed in the release notes.

A convenient moment is right after a campaign has landed and its gallery pages are frozen — the tests are fresh in memory and the release notes write themselves from the campaign summaries.

2. Phase A — freeze and tidy¶

cd ~/Codes/suite
tools/release/check_release_state.sh          # versions, tags, pins, PyPI, worktrees, behind/ahead

A1. Merge or park every worktree branch (§1.1). Remove finished worktrees: git -C flow worktree remove ../flow-<name> (after the branch is merged and pushed).
A2. Fast-forward every checkout: git -C pull --ff-only for each of core morton flow pnm dem voro coupling, then git submodule status must show no +.
A3. Scratch hygiene. The umbrella working tree carries untracked campaign directories (flow-* worktrees; sphere-cfd-validation/ and proposal/ moved to ~/Codes/ on 2026-09-08, and on 2026-09-22 .sdf-campaign-probes/ into flow/tests/study/sdf_campaign/ and the hash-named paper repo to ~/Codes/paper-robust-scaled-ibm-solver, leaving the worktrees as the only untracked scratch). They are not committed and do not affect the release, but the sdist of a submodule built while the repo is a submodule (.git is a file) tree-walks and pulls in build//.venv. Never build a release sdist locally from the submodule checkout — CI builds from a real clone. If you must build locally, clone the repo fresh.
A4. Stale test/scratch files inside submodules (voro/test_polymesh_sphere.vtu, voro/python/__pycache__, dem root-level verify_*.py logs): delete or .gitignore; never git add -A (stage named paths only).
A5. Confirm CI green everywhere (§1.4). Dependabot action-bump PRs need gh auth refresh -s workflow to merge; merge them before the freeze, not during.

3. Phase B — fully functioning codes (the test matrix)¶

Run on the workstation with OMP_NUM_THREADS=8 OMP_PROC_BIND=false (the CUDA prefix has an OpenMP host backend; an unbounded pool on 48 cores is an hour-long trap) and with PATH=/usr/local/cuda-13.2/bin:$PATH. Build trees named build_rel_* so they cannot be confused with campaign trees; delete them first — FindPython artefact variables are sticky and a tree configured against a retired venv fails with "Cannot run the interpreter".

Project	Suite	Command (host prefix `extern/install/host-openmp`; repeat with `nvidia-cuda`)	Expect
core	serial + MPI halo, migration, load balancing	`cmake -S core -B core/build_rel -DCMAKE_PREFIX_PATH=$P && cmake --build core/build_rel -j && ctest --test-dir core/build_rel --output-on-failure`	all pass, np 1–8
core	Kokkos device halo	`cmake -S core -B core/build_rel_k -DPECLET_CORE_ENABLE_KOKKOS=ON ...` per `core/CLAUDE.md`	all pass
amr	octree, AMR NS, MPI (np 1–8)	`cmake -S amr -B amr/build_rel -DCMAKE_PREFIX_PATH=$P && ... ctest` per `amr/CLAUDE.md`	all pass
morton	doctest + Kokkos + pytest	`ctest --test-dir morton/build_rel`; `python -m pytest morton/bindings/python/tests -q`	all pass; non-BMI2 binary PDEP/PEXT-free
flow	kernel unit tests	`cmake -S flow/tests/kokkos -B flow/build_rel_kokkos -DCMAKE_PREFIX_PATH=$P && ... ctest`	all pass
flow	distributed (`PECLET_FLOW_MPI`)	`cmake -S flow/tests/kokkos_mpi -B flow/build_rel_kmpi -DMPIEXEC_EXECUTABLE=/usr/bin/mpirun ...`	np 1,2,4 all pass, np=1 bit-exact
flow	accuracy/efficiency regression	`PYTHONPATH=flow/build_rel python flow/tests/regression/sdflow_regression.py`	within baseline; do NOT `--update` to make it pass
flow	analytic verifications	`flow/scripts/verify_poiseuille_flow.py`, `verify_periodic_spheres_sdflow.py`, `verify_lid_cavity_sdflow.py`, `verify_channel_sdflow.py`, `verify_bfs_sdflow.py`	each prints PASS
pnm	extraction + MPI	`pnm/scripts/test_extraction.py ../flow/data/packing_ring.vti`; `tests/kokkos_mpi` np 1,2,4	7199 pores; bit-exact
dem	kernel + MPI	`dem/tests/kokkos` (8), `dem/tests/kokkos_mpi` (24, host + CUDA); `python dem/verify_packing_spheres.py`	all pass
voro	Kokkos + MPI	`cmake -B voro/build_rel -DPECLET_VORO_KOKKOS=ON -DPECLET_VORO_BUILD_PYTHON=ON [-DPECLET_VORO_MPI=ON]`; `OMP_PROC_BIND=false ctest`	all pass (a plain build registers zero tests — not a pass)
coupling	Python integration tests	`cmake -S coupling -B coupling/build_rel -DPECLET_COUPLING_TESTS=ON -DPECLET_FLOW_BUILD=<flow tree> -DPECLET_DEM_BUILD=<dem tree> ...; ctest` — both `_BUILD` variables are required, see below	all pass
whole family	import smoke	`PYTHONPATH=<all build dirs> python -c "import peclet.flow, peclet.dem, peclet.voro, peclet.pnm, peclet.core.mpi, peclet.morton, peclet.coupling"`	every `execution_space` reports the intended backend

Record the counts in RELEASE_PREP.md (they become the "tested" line of the release notes). Anything red is a release blocker unless it is documented as a known limitation.

Free MPI ctests in CI: only the OpenMP single-rank builds run on GitHub (no GPU, no MPI runners). The MPI and CUDA columns of this matrix are a local obligation — the release checklist is the only place they are enforced.

coupling's two _BUILD variables are load-bearing, and omitting them fails SILENTLY. coupling/tests/CMakeLists.txt builds the ctest PYTHONPATH as "${CMAKE_BINARY_DIR}:${PECLET_FLOW_BUILD}:${PECLET_DEM_BUILD}". Leave them unset and those two entries are empty, so the test subprocess falls through to whatever peclet-flow / peclet-dem are installed in the venv — which until 2026-09-16 were 0.4.0 wheels from July (now uninstalled, see the note below; today the same omission raises ModuleNotFoundError instead). Measured 2026-09-12: the bare configure line makes 2 of 3 coupling tests abort with TypeError: initialize_shape(): ... shape_type: int and a matching set_body_force() error, i.e. the July signatures, not the current ones. That is the lucky outcome. The dangerous one is the same fall-through when the old signature still happens to match: the tests pass, against a module nobody intended. This is the same failure class as RELEASE_PREP 1.0–1.1 §1.3's SDFLOW_BUILD trap, and it is worth stating the general rule: anything in this checklist that imports peclet.* must set PYTHONPATH to the build trees explicitly, or it is not testing what you think it is. That includes tools/gen_python_api.py and tools/release/audit_docstrings.py.

The silent half of this trap was removed on 2026-09-16. .venv used to carry stale installed peclet wheels (peclet 0.6.0, peclet-flow 0.4.0, peclet-dem 0.4.0, peclet-voro 0.4.0, peclet-pnm 0.1.0 — July builds), so a forgotten PYTHONPATH fell through to those instead of failing. They are uninstalled and site-packages carries no peclet residue, so a bare import peclet.flow now raises ModuleNotFoundError — loud, and immediately diagnosable — instead of quietly testing code from July. Do not reinstall them into .venv; develop against build trees, and use a throwaway venv to test a published wheel.

Two consequences worth knowing. A build-tree import reports __version__ == "0+unknown", because the version comes from importlib.metadata and a build tree has no dist-info — that string is now the signal that you are correctly on a build tree, not a defect. And the family import smoke at the foot of the matrix can no longer report a plausible-looking wrong version: its version column was never trustworthy while those wheels were installed, which is why the criterion is execution_space, not the version.

4. Phase C — versions¶

4.1 Decide the numbers¶

Semantic versioning per package; the family version is the metapackage's. Rules used so far:

major bump when a package removes or renames public Python API. The first one is the family's 1.0.0 (QUALITY_PLAN.md D9): every package ships 1.0.0 together — except peclet-amr, which ships 0.1.0 and stays 0.x while under development (D9's recorded exception) — the umbrella CHANGELOG lists every removed name with its replacement, and from then on the NAMING.md alias ladder is binding, so a further break costs a major.
minor bump when a package gains public Python API or a numerical method (VoF, moving geometry, a new drag law). (Before 1.0.0 a removal was also a minor — flow 0.4.0 removed peclet.flow.pnm.)
patch bump for fixes with no API change.
A package with zero commits since its tag keeps its version (morton and pnm at 0.6.0 time); it is not re-released, and its == pin in the metapackage stays.
The metapackage always bumps (its pins changed).

4.2 Where versions live (bump every one, in this order)¶

File	Field	Notes
`<sub>/pyproject.toml`	`version`	the single source (QUALITY_PLAN D4): CMake's `project(VERSION)` reads it at configure time, so no CMake literal to bump; tag must equal `v<version>`
`<sub>/packaging/*_init.py` (`__version__`) and `morton/bindings/python/peclet/morton/__init__.py`	`__version__`	must match pyproject (0.2.0 shipped with a stale core `__version__`)
`flow/packaging/pyproject-cuda.toml` (and each future `*-cu13`)	`version`	same number as the CPU package
`<sub>/Doxyfile`	`PROJECT_NUMBER`	Doxygen sites show it
`<sub>/CITATION.cff`, umbrella `CITATION.cff`	`version`, `date-released`	Zenodo reads these
`<consumer>/cmake/PecletDeps.cmake`	`PECLET_TPX_TAG`, `PECLET_MORTON_TAG`	flow, pnm, dem, voro, coupling: pin to the new core/morton tags — voro shipped 0.4.0 pinned to a stale core tag
umbrella `pyproject.toml`	`version`, every `==` pin, extras	publish the members first — the pins must resolve on PyPI
`docs/containers.md`, `containers/README.md`, `containers/submit/*.slurm`	image tags in the `apptainer pull` lines	today they still say `0.1.0`
`CHANGELOG.md`	new section	Keep-a-Changelog format; one section per family version

tools/release/check_release_state.sh prints all of these next to each other so a mismatch is visible before tagging.

5. Phase D — documentation and the Python API¶

5.1 Docstrings and generated API pages¶

Build the whole family from MPI-enabled builds (host prefix is enough) and put every build tree on PYTHONPATH. Fix docstring findings in the binding TUs (/src/*_bindings.cpp, core/python/*_bindings.cpp); tools/release/audit_docstrings.py --strict lists bound callables without a docstring (a failure) and docstrings that still say sdflow/vorflow/tpx/ peclet.flow.pnm. check_release_state.sh runs it: a missing docstring blocks the release.
The Python API pages (docs/python/<pkg>.md) are not committed and not regenerated by hand. The Site workflow generates them at build time with tools/gen_python_api.py, run inside the released peclet-cpu image; after the tag, containers.yml dispatches the Site workflow with the pinned version once that image is pushed (strict: every PAGES module must import, from an MPI build). So the pages describe the released API, and the docstrings fixed in step 1 reach the site with the release. What the pre-flight still checks is coverage: check_api_pages.py fails when a package pinned by the metapackage (pyproject.toml) lacks a PAGES entry in tools/gen_python_api.py or a mkdocs.yml nav entry. PAGES covers every shipped package (peclet.flow/pnm/dem/voro/coupling/amr/morton, peclet.halo + peclet.geom on core.md; peclet-core is the deprecated shell and is covered by peclet.halo). A new package must also be installed in containers/cpu.def, or its page is a "not importable" stub.
mkdocs build --strict must pass (the Site workflow runs exactly that). Locally, generate the pages first with tools/gen_api_local.sh (Apptainer; pulls peclet-cpu:latest), then mkdocs serve.

5.2 Prose¶

Update, in this order, and read each one end to end (they drift in small factual ways):

umbrella README.md, docs/index.md, docs/DEPLOYMENT.md, docs/containers.md, docs/PORTABILITY.md, docs/SNELLIUS.md, docs/LUMI.md, containers/README.md;
each submodule README.md (badges, install line, feature list) and CLAUDE.md build/test sections;
CHANGELOG.md — write the family section from the per-package git log <old-tag>..HEAD --oneline and the campaign summaries in docs/*_PLAN.md;
CLAUDE.md links.
The landing pages are PyPI metadata, and PyPI metadata is IMMUTABLE. There is no "edit README" on pypi.org: the long description is fixed at upload time. A stale README noticed after a tag can only be corrected by publishing another version, which is why peclet 1.1.1 exists ("metapackage only, to refresh a page no edit could reach") — and 1.2.0 still shipped three pages (peclet, peclet-cu13, peclet-halo) describing the very layout that release replaced.

tools/release/check_landing_pages.py is the gate, and it runs in two places so it cannot be skipped: check_release_state.sh runs it over the whole suite in the pre-flight, and the umbrella's release.yml runs it as a landing-pages job that publish needs — the last moment the mistake is still cheap.

The CI job passes --dists peclet,peclet-cu13: only the pages that upload CREATES. A member's PyPI description is created by that member's own release, and CI checks submodules out at the umbrella's recorded pointers, which legitimately lag a member's main whenever that member is deliberately not being re-released. Without the scope, a metapackage refresh is blocked by a page it does not publish — which is exactly what happened on 1.2.1's first attempt: the job failed on dem/voro/pnm/coupling READMEs that were already fixed on their mains. Nothing was uploaded (the gate ran before publish, so the version was not burned) and the tag was re-cut after scoping. Every member repo now has the same job (landing-pages, which its publish needs), added 2026-09-21. A member does not vendor the checker: its job sparse-checks /docs/NAMING.md and /tools/release/check_landing_pages.py out of the umbrella's main and runs

python .peclet-umbrella/tools/release/check_landing_pages.py \
       --naming .peclet-umbrella/docs/NAMING.md \
       --page README.md --dists peclet-flow,peclet-flow-cu13

so nine repos cannot drift apart on what counts as a retired spelling — the canon has one home. --dists is per repo and lists every distribution that README is the description of, which is how the *-cu13 pages get covered: flow/README.md is the PyPI page of both peclet-flow and peclet-flow-cu13, and it used to name neither.

It needs no list of its own. docs/NAMING.md §2 is the canon; the tool parses every | former | canonical | status | row whose status is removed/aliased/planned and greps the landing pages for the former spellings. So every rename is covered the day it is recorded in NAMING.md, which is one more reason to record it there. It also checks that a package's page names the distribution it actually ships (core's README badged peclet-core while publishing peclet-halo).

Naming an old spelling is not itself a failure — good migration prose should ("was peclet.core.geom until 1.2.0"). The rule is that the canonical replacement must appear within three lines. For a deliberate mention with no replacement to name — a negative statement like "the module has no mpi_rank()" — mark it . Repository URLs, badge URLs, Zenodo DOIs and find_package(...) are exempt: the peclet-core repository did not move, only the distribution did.

No version literals on the site pages (docs/index.md, DEPLOYMENT.md, containers.md, python/index.md, umbrella README.md): the PyPI badge, the CHANGELOG and the moving container tags (latest / sm80 / sm90 / gfx90a) carry the current version, and CITATION.cff keeps only the concept DOI — so nothing on the site goes stale between releases (the "This release (v0.2.0)" line survived five releases before this rule). tools/release/check_release_state.sh greps for it.
The quick start must RUN against the published wheels, not merely parse — and CI owns this. .github/workflows/quickstart.yml executes README.md, docs/index.md and docs/notebooks/quickstart_sphere.ipynb against the wheels on PyPI, and matches every python block under docs/ and every submodule README.md against the installed surface. It runs on a docs push or PR, weekly (the docs also break when the API moves under them, with no docs commit to trigger on), and — as a quickstart job of release.yml, needs: publish — on the wheels of every new tag, retrying while PyPI propagates. A release is not done until that job is green.

Locally it is the same tool, tools/release/check_docs_snippets.py:

python tools/release/check_docs_snippets.py                    # names at v1.0.0, from the submodules
python tools/release/check_docs_snippets.py --installed --run  # exactly what CI runs, on the wheels

Before a tag exists the --ref form is the only one available (--installed can only ask for what is already published), so run it against the submodule HEADs while preparing the release and let the tag-triggered job confirm the real artifact.

Why it exists: 1.0.0 shipped a landing page whose first 20 lines raised TypeError. The clean break had renamed cell_centres() to cell_centers() and repacked set_body_force(fx, fy, fz) into one 3-sequence; audit_examples.py covers only the sibling gallery, so nothing looked at the suite's own docs, and nothing at all executed them. Note which check catches which: the rename is a NAMES failure, but the repacked signature renamed nothing and is invisible to any static audit — only running it finds that class.

The audit list for the current release is in RELEASE_PREP.md.

5.3 CUDA wheel coverage (decision point)¶

The CUDA family is COMPLETE (re-verified 2026-09-12; this paragraph used to say only peclet-flow-cu13 existed and list the rest as future work). flow, pnm, dem and voro each carry a cuda-wheel job and a packaging/pyproject-cuda.toml, and all four are live on PyPI — peclet-flow-cu13 0.5.1, peclet-pnm-cu13 0.1.2, peclet-dem-cu13 0.5.1, peclet-voro-cu13 0.5.1 — beside the peclet-cu13 metapackage at 0.7.2 (umbrella packaging/pyproject-cu13.toml, published by a second job of the umbrella release workflow). The operational consequence: there are FOUR CUDA version literals to bump, not one, and each must match its CPU package's number. dem additionally builds ArborX into the prefix, and each module's rpath is $ORIGIN/../../nvidia/cu13/lib, one level per package depth — pnm's was wrong once. Per-package prerequisites: the module must survive interpreter teardown after a solve (dem's atexit-finalize pattern; flow had a teardown abort), and the PyPI Trusted Publisher must be registered for the new project name before tagging (a new name publishes fine with no extra registration — pnm 0.1.0 proved it — but only if the workflow's environment: pypi is configured).

Validate locally before CI: PATH=/usr/local/cuda-13.2/bin:$PATH CMAKE_PREFIX_PATH=$PWD/extern/install/nvidia-cuda pip wheel ./dem into a fresh venv, then run a verify script on the GPU.

6. Phase E — publish to PyPI¶

Per repo, in dependency order (§0). For each:

cd <sub>
# 1. version bumps (§4.2) committed on main, pushed
git push origin main
# 2. tag == pyproject version
git tag v<X.Y.Z> && git push origin v<X.Y.Z>
# 3. watch the Release workflow (sdist + wheels [+ cuda-wheel] + publish)
gh run watch -R computational-chemical-engineering/peclet-<sub> $(gh run list -R computational-chemical-engineering/peclet-<sub> -w Release -L 1 --json databaseId -q '.[0].databaseId')
# 4. confirm on PyPI (the JSON endpoint caches for minutes — use pip)
pip index versions peclet-<sub>

Then the umbrella: bump pyproject.toml pins + version, bump the submodule pointers (git add core flow ... by name), commit, push, tag v<family>, watch Release and Containers, then §9.

Traps recorded from 0.1.0–0.6.0:

cibuildwheel 4.x dropped cp38: keep requires-python >= 3.10 and the build selector in sync.
macOS Intel runners queue for 40+ minutes and gate publish — morton dropped macos-13; do not add it back.
skip-existing: true in the flow publish step lets a partial upload be completed (e.g. adding the cu13 wheels after the CPU files); leave it.
A publish failure whose traceback ends in sigstore/_internal/rekor/client.py is TRANSIENT INFRASTRUCTURE, not your release. pypa/gh-action-pypi-publish signs an attestation against Sigstore's Rekor transparency log before uploading anything, so a ConnectionError: Connection reset by peer there means nothing reached PyPI. Hit 2026-09-12 on the umbrella's own publish job with both build jobs already green. gh run rerun <id> --failed fixed it on the first retry. Do NOT respond by editing versions or cutting a new tag — skip-existing: true makes the retry safe even if part of an upload had landed.
A pypi/pyversions badge renders "missing" without trove classifiers; the static python badge is deliberate.
CUDA wheels must carry native SASS for every GPU major you support (sm_75/80/90/100/120 today) — PTX does NOT carry a wheel forward: a driver older than the toolkit that emitted the PTX cannot JIT it, and under minor-version compatibility the launch silently no-ops, so Kokkos aborts at import with "likely mismatch of architecture" (0.4.0–0.5.0 shipped sm_75 + CUDA-13.2 PTX only and failed on every non-Turing GPU with a 13.0/13.1 driver). The extra SASS goes in through NVCC_APPEND_FLAGS (nvcc reads it itself; nvcc_wrapper rejects a second -arch/-gencode), and the job pins the OLDEST 13.x toolkit so the PTX JITs on any 13.x driver. Check a wheel with cuobjdump --list-elf <module>.so.
A gallery page with a stale freeze does not fail loudly. Its bootstrap pip-installs peclet and re-executes on CPU wheels, so a §9 re-check can quietly produce CPU-rendered numbers instead of a clean red — the failure mode is a slow silent re-execution, not an error. Reported from the peclet-examples side 2026-09-14, where a stray untracked draft installed peclet into a deliberately peclet-free gate venv.
Smoke test in a fresh venv after the metapackage is live: python -m venv /tmp/rel && /tmp/rel/bin/pip install peclet==<family> && /tmp/rel/bin/python -c "import peclet.flow, peclet.dem, peclet.voro, peclet.pnm, peclet.morton", then run one gallery example (poiseuille-ibm) against it; repeat with peclet-cu13 on the GPU box AND peclet[mpi,cfd-dem] (the sdist members: core, coupling — the core sdist was silently unbuildable 0.1.0–0.6.0).

7. Phase F — Snellius package¶

What "a release for Snellius" means, in decreasing order of user convenience:

The GHCR CUDA images peclet-cuda:<family>-sm80 (A100) and -sm90 (H100), pulled with apptainer pull oras://... and launched by containers/snellius-run.sh / containers/submit/snellius.slurm. Multi-node runs need the host OpenMPI/UCX bind (the image bundles OpenMPI 4.1.6; match the site module).
A site-built install in the project space, from the release tag, with the validated 2024a toolchain (docs/SNELLIUS.md): tools/hpc/install_snellius.sh <tag> h100|a100|cpu. It clones the umbrella at the tag, bootstraps Kokkos for the backend, and pip-installs the whole family (PECLET_*_MPI=ON; distributions peclet-morton peclet-geom peclet-halo peclet-core peclet-flow peclet-pnm peclet-dem peclet-voro peclet-amr peclet-coupling) into $SUITE/.venv, leaving a wheelhouse at /projects/0/prjs1022/peclet/wheelhouse/<tag>-<backend>/ so project members can pip install --no-index --find-links <wheelhouse> peclet-flow into their own venv without rebuilding. Wheels built on Snellius are site-specific (linked against the module OpenMPI, CUDA 12.6, sm_80/sm_90) and must never be uploaded to PyPI.
Source build by the user following docs/SNELLIUS.md (the fallback; unchanged).

Release steps (billed: one build job per backend ≈ 1 GPU × ≤2 h; queue the CPU build on genoa):

ssh snellius
cd /projects/0/prjs1022/peclet && git clone --branch v<family> --recurse-submodules https://github.com/computational-chemical-engineering/peclet.git suite-v<family>
cd suite-v<family>                       # SUBMIT FROM INSIDE THE TREE -- see the two traps below
sbatch --nodes=1 --gpus-per-node=1 --ntasks-per-node=1            tools/hpc/install_snellius.sh v<family> h100
sbatch -p gpu_a100 --nodes=1 --gpus-per-node=1 --ntasks-per-node=1 tools/hpc/install_snellius.sh v<family> a100
sbatch -p genoa --gpus-per-node=0 --ntasks=1 --cpus-per-task=32    tools/hpc/install_snellius.sh v<family> cpu

Two traps, both hit on this script's first real run (2026-09-16, the 1.1.0 release), both now fixed in the script — the recipe above is the corrected one.

Submit from inside the tree. sbatch copies the script to /var/spool/slurm/..., so $BASH_SOURCE is not in the repo and the $SLURM_SUBMIT_DIR fallback is what resolves snellius_env.sh. Submitting from the parent (which this file used to say) resolved to $PROJ/tools/hpc and died with a bare No such file or directory. The script now tries three candidates and, failing all of them, prints what it tried and the fix.
Each backend gets its own tree (suite-<tag>-<backend>). h100 and a100 both build the nvidia-cuda prefix but at different arch, and the script does venv --clear on $SUITE/.venv and rm -rf extern/install/$BACKEND — so two backends in one tree destroy each other whether they run together or in sequence. Submitted together against one tree, two of three died on a half-built venv (Permission denied: .../.venv/bin/activate.csh) and the survivor would have left a venv matching only itself. With per-backend trees the three run concurrently.
--gpus=0 does not override the script's #SBATCH --gpus-per-node=1 on a GPU-less partition; use --gpus-per-node=0 or sbatch rejects the job with "Requested node configuration is not available".

Then the smoke job tools/hpc/smoke_snellius.slurm (4 ranks on one node: has_mpi: True, execution_space: Cuda, flow init_mpi bit-exact k vs np=1, dem step_mpi) and record the job ids + numbers in docs/SNELLIUS.md "Validated releases". Give the release its own remote tree (suite-v<family>), never the shared campaign checkout — concurrent sessions have rebuilt the shared build_cuda_mpi on top of each other's uncommitted sources.

Snellius rules that apply here: positional sbatch arguments (leading VAR=x sbatch is silently dropped), --gpus-per-node=1 for a 1-GPU job (billing is per allocated GPU), -DPython_EXECUTABLE with a capital P, -DMPIEXEC_EXECUTABLE=$(which mpirun), FRESH=1 when the CUDA module changed.

8. Phase G — LUMI package (untested)¶

State: no peclet-hip image has ever been published. The hip-gfx90a container job fails at link time in every release run since 0.3.0 with

ld.lld: error: undefined hidden symbol: vtable for Kokkos::Impl::SharedAllocationRecord<Kokkos::HIPSpace, ...>
ld.lld: error: undefined hidden symbol: vtable for std::_Sp_counted_ptr_inplace<peclet::core::halo::GridHaloTopology<3>, ...>

-Wl,--no-gc-sections (present in flow/pnm/voro/coupling CMake since 0.2.0) did not fix it, so the diagnosis "gc-sections drops the vtables" is refuted. The references are emitted with hidden visibility because nanobind sets CXX_VISIBILITY_PRESET hidden on the module target (nanobind_set_visibility), and under hipcc/lld the vtables of these header-instantiated classes are not emitted in the same object. The same link will fail for a source build on LUMI, so this blocks every LUMI delivery, container or not.

Resolved 2026-09-04: experiment 1 (run 33868865327) made flow, dem, voro link; core's amr and voro's wrappers then failed for virtual classes in anonymous namespaces (named namespaces fixed it); run 33874788583 built the full image. The rule for new binding code: virtual wrapper classes go in a named namespace. The list below is kept for the record.

Experiments, in order of plausibility (each is one workflow_dispatch of Containers with only=hip, push=false, ≈25 min; no GPU is needed to link):

After nanobind_add_module(...) in flow/pnm/dem/voro/coupling CMakeLists.txt, gated on Kokkos_ENABLE_HIP: set_target_properties(<target> PROPERTIES CXX_VISIBILITY_PRESET default).
Bootstrap Kokkos for lumi-hip as a shared library (-DBUILD_SHARED_LIBS=ON in tools/bootstrap_deps.sh), so the Kokkos vtables come from libkokkoscore.so.
Relocatable device code: -DKokkos_ENABLE_HIP_RELOCATABLE_DEVICE_CODE=ON + -fgpu-rdc on the modules.
Drop -fvisibility-inlines-hidden if the ROCm clang adds it; check hipcc --verbose.

Until one passes, the LUMI deliverable of a release is:

docs/LUMI.md — the site recipe (Cray PE modules, hipcc, craype-accel-amd-gfx90a, ROCm ≤ host driver, Cray-MPICH ABI bind model) with a banner "untested — no LUMI allocation yet";
tools/hpc/install_lumi.sh — the source-install script mirroring the Snellius one (untested);
containers/hip.def + lumi-run.sh + submit/lumi.slurm as they are, with the known-failure note in docs/containers.md.

The release notes must say this in one sentence. When LUMI time is granted: run experiment 1 on a LUMI login node first (link only, seconds), then the smoke job, then re-dispatch the container job.

9. Phase H — containers, GitHub Releases, Zenodo, sites¶

Containers: the umbrella tag triggers containers.yml (cpu, cuda-sm80, cuda-sm90 ≈ 20 min each; hip fails — expected until §8 is fixed, and it does not block the others because the matrix is fail-fast: false). Verify: apptainer pull oras://ghcr.io/computational-chemical-engineering/peclet-cuda:<family>-sm90 on Snellius and run the smoke job through containers/snellius-run.sh. The images install flow, dem, voro, core, morton only — add pnm and coupling to cpu.def/cuda.def/hip.def (§ RELEASE_PREP) so the container family matches PyPI.
GitHub Releases: one per repo at its new tag + the umbrella, body = the CHANGELOG section (gh release create v<X> -R computational-chemical-engineering/peclet- --title "peclet- <X>" --notes-file <notes>). Zenodo mints a version DOI per repo within minutes; the concept DOIs in the badges and CITATION.cff do not change; CITATION.cff carries only the concept DOI (a version DOI there goes stale — Zenodo and each GitHub release link the version's own).
Sites: the umbrella Site workflow runs on the docs push; each repo's docs.yml on its push. Check that the PyPI badge on https://computational-chemical-engineering.github.io/peclet/ shows the new version (shields.io caches for a few minutes) and that the Python API pages carry the new methods: the peclet-cpu leg of containers.yml dispatches the Site workflow with the pinned image tag (§5.1; there is no version to edit on the site).
Memory/notes: update docs/SNELLIUS.md "Validated releases", RELEASE_PREP.md → archive the closed cycle's sections verbatim as docs/archive/RELEASE_PREP_<cycles>.md (entry in docs/archive/README.md), and reset RELEASE_PREP.md for the next cycle.

10. Phase I — the example gallery (after the release, not blocking it)¶

The gallery pins nothing: every bootstrap cell does pip install peclet (latest) unless PECLET_LOCAL_BUILD points at a build tree, and the site renders from committed _freeze/ outputs. So a release cannot break the published site, but it can make a page's Colab button run a different solver than the frozen numbers came from. The obligations are therefore:

Before tagging, produce the re-check list with tools/release/audit_examples.py (or the table in RELEASE_PREP.md): for each page, the peclet symbols it calls and whether each exists in the tagged bindings. Pages calling symbols that only exist on main were rendered against a local build and become runnable on PyPI only with this release — they are the ones to re-execute first, because their frozen output was produced by a pre-release build.
After the metapackage is live: re-execute the list in priority order in a fresh venv (pip install peclet==<family>; for GPU pages peclet-flow-cu13), page by page with render_example.sh <slug> (clears the page's freeze) — note that editing even a subtitle invalidates a freeze, and quarto exits 0 on a figure-cell error, so grep _site for cell-output-error before committing. Commit each page's _freeze/ (force-add npz assets; a movie does not go in the repository — see §10.1).
Pages that need a build the wheels do not provide (MPI, Snellius, peclet.coupling, core.geom) stay frozen and say so in their bootstrap comment; they are re-checked when the corresponding site package (§7) is installed.
Close or update the ISSUES.md entries the release fixes; open new ones for what the re-run finds.

10.1 The films and the channel¶

Movies live on @PecletHPC, never in a git repository, and ~/Codes/youtube_publishing (ytpub) is the whole workflow: videos.yaml is the hand-written registry, published.json the ledger of slug → video id that makes a second run an update rather than a duplicate upload. Quota is 10 000 units a day — an upload is 1600, a metadata or privacy change 50 — so plan a release's uploads, and do not re-run sync idly.

cd ~/Codes/youtube_publishing
./ytpub.sh doctor                     # token, ffmpeg, manifest, and which channel it points at
./ytpub.sh sync --dry-run             # what would happen and what it costs
./ytpub.sh sync <slug>                # upload or update one
./ytpub.sh embed --write <slug>       # rewire the example page to the YouTube copy
./ytpub.sh channel --apply            # description, keywords, banner, trailer

Per release:

Cut the release film in films/<version>/ (tools/narrate.py then tools/assemble.py), and watch it end to end before it goes anywhere. The audio and the picture are cut from one pause constant in films/voice.yaml; if they ever drift, the film ends mid-sentence and -shortest silently truncates the close.
Publish it: an entry in videos.yaml with root: repo, encode.passthrough: true (the film is already 1080p30 and carries its own narration) and page_url: pointing at the documentation, then ytpub sync <slug>. New clips default to privacy: unlisted — watch each one on the channel, then set privacy: public and re-run sync (50 units, no re-upload).
Make it the front door: channel_trailer: in videos.yaml and ytpub channel --apply. The avatar is the one thing with no API — YouTube Studio only.
Rewire the pages: ytpub embed --write <slug> for any page still carrying a local <video>, then git rm --cached the mp4. --freeze can mirror the swap into Quarto's freeze cache and re-stamp its hash, but only use it on a page whose frozen output is genuinely current — during a re-render pass (§10 step 2) the pages are re-executed anyway and a re-stamp would publish stale output as if it were fresh.
Retire the previous film to unlisted. Set the previous release's entry to privacy: unlisted and sync it — do not delete the entry, and do not use private. The entry is what holds the video id, and sync only pushes a privacy it can read from the manifest, so deleting it leaves the old film public with nothing left to notice. Why unlisted and not private: a published distribution carries a FROZEN copy of the README and PyPI will not let a version be re-uploaded, so every release artifact links to the film of its day immutably. Private turned the peclet 1.1.0 page into a 404 thumbnail and a sign-in wall on 2026-09-19 — the page pip install peclet sends people to. Unlisted keeps those references alive while keeping the film off the channel and out of search. Reserve private for something that should never have been published. Then move channel_trailer: to the new slug and ytpub channel --apply.
Interlink both ways: every video description carries its page, the repository and the docs (site.docs in videos.yaml); the docs home page, the gallery landing page and the README carry the film and the channel. Swapping a film means swapping the id in all three — and in the gallery's landing page the embed MUST stay inside its ``{=html} fence, or Pandoc wraps the iframe in a
and Bootstrap's.ratio` renders it at 300x150.

Picture quality is films/QUALITY.md, and it is a design contract, not advice: the 2160p master, perspective rather than zoompan for the push-in, one x264 encode per item, and the BT.709 conversion. tools/quality_check.py film gates all four on every cut. The 1.0.0 film shipped with a shimmering push-in and the channel teal displayed as (0,132,136); both were invisible to every check that existed at the time.

11. CI on GitHub — what runs, what must be green, what is missing¶

Repo	Workflows	Trigger	Runs
core	`ci.yml`, `quality.yml`, `docs.yml`, `release.yml`	push/PR; tag	OpenMP build + ctests, serial and MPI via mpirun on the runner
amr	`ci.yml`, `quality.yml`, `docs.yml`, `release.yml`	push/PR; tag	the eighth package (split out 2026-09-10); its `ci.yml` carries an explicit "Honest CI" header and is the best-documented in the suite
morton	`ci.yml` (gcc, clang, AVX-512 under SDE, MSVC, python, docs), `quality.yml`, `docs.yml`, `release.yml`	push/PR; tag	the most complete matrix in the suite
flow	`ci.yml` (`single-rank` + `mpi`), `quality.yml`, `docs.yml`, `release.yml` (sdist, wheels, cuda-wheel, publish)	push/PR; tag	single-rank AND distributed at np = 1, 2 (4 and 8 when the np=2 pass left time)
pnm	`ci.yml`, `quality.yml`, `docs.yml`, `release.yml`	push/PR; tag
dem	`ci.yml`, `quality.yml`, `docs.yml`, `release.yml`	push/PR; tag	single-rank only
voro	`ci.yml` (kokkos-openmp, clang-format, documentation), `quality.yml`, `docs.yml`, `release.yml`	push/PR; tag	its pinned-tag guard COMPILES the module since `8cdac25`, which is what makes a stale `PECLET_CORE_TAG` fail here rather than during the release
coupling	`ci.yml`, `quality.yml`, `release.yml`	push/PR; tag	builds flow + dem + coupling and runs the pytest battery
umbrella	`site.yml`, `release.yml`, `containers.yml`, `quickstart.yml`	docs push; tag; dispatch; weekly	`quickstart.yml` executes the landing page and the Colab notebook against the PyPI wheels, and runs as a `needs: publish` job of `release.yml` on every tag (added 2026-09-12, after 1.0.0 shipped a quick start that raised `TypeError`)
peclet-examples	`publish.yml`	push to main	Quarto render from freeze (no solver)

(Table re-verified 2026-09-12 against .github/workflows/ in every repo. It previously predated amr, said coupling had "no CI at all" and pnm "no quality.yml", and described flow's CI as single-rank only — all four were false.)

Must be green before tagging: every ci.yml and quality.yml python-lint job on main. The clang-format jobs are informational by design.

Gaps to close (tracked in RELEASE_PREP.md; none blocks a release, all reduce risk):

Containers push gate — add a push boolean workflow_dispatch input to containers.yml so HIP experiments (§8) can build without publishing an image tagged with a wrong version.
A family "release rehearsal" workflow in the umbrella (workflow_dispatch): checks out all submodules, builds the CPU wheels of every member with cibuildwheel into artifacts without publishing, installs them together in one venv and runs the import smoke + one verify script. This is the only automated place where the family is tested as a set before tagging.
Version consistency check as a CI step on tags (tools/release/check_release_state.sh --ci): fail the release job if the tag ≠ pyproject version or __version__.
~~The CUDA wheel job runs only on tags~~ — wrong, and expensively so. Not one of the four cuda-wheel jobs (flow, pnm, dem, voro) carries an if:; only publish is tag-gated. So the dry run this bullet asks for already works today: gh workflow run release.yml on any of them builds the CUDA wheel and uploads it as an artifact, publishing nothing. Nobody knew, so nobody rehearsed, and on 2026-09-14 voro's cuda-wheel job died at GitHub's 6-hour cap during the 1.0.1 release — the one job a rehearsal would have exercised. Corrected after the fact; see RELEASE_PREP 1.0–1.1 §11.5.
No release.yml in any of the eight repos sets timeout-minutes, so every job inherits the 6-hour default. That is not the cause of voro's failure — the work genuinely takes ~5 h — but it means the cap arrives as a kill with no warning rather than as a stated budget.
actions/upload-artifact / download-artifact majors differ between repos (v4 vs v7/v8) — the Dependabot bumps must be merged consistently or the artifact hand-off between jobs breaks.

12. Execution advice¶

Run it as a two-day block, not interleaved with development: day 1 = Phases A–D (tests, docs, versions) on the workstation; day 2 = Phases E–H (publish, containers, Snellius). Snellius builds queue for up to an hour — submit them first thing on day 2 and do the PyPI publishing while they run; they only need the tag to exist.
One driver, one checklist. Copy the phase headers of this file into RELEASE_PREP.md as a checkbox list and tick them in the order given; parallel agents may prepare docs and tests, but the tag/publish steps are sequential by dependency and should be done by one session.
Tags are permanent. If a release workflow fails after the tag exists, fix and re-run the workflow (gh run rerun --failed); never delete and re-push a tag that reached PyPI. A broken published package gets a patch release.
Do not touch numerics during the release window. Version bumps, docs and packaging only; a numerical fix found in Phase B is either a blocker (fix, re-run the matrix from the top) or a known limitation (release notes).
Expected wall time from tagging the first member to a live metapackage: about 90 minutes (the CPU wheel builds compile Kokkos per Python version; the CUDA job is the long pole at ~40 min).