ADR 0032: The CUDA plugin wheels in CI and in the release workflow#

  • Status: Accepted under delegation (2026-10-06; GOVERNANCE.md, “Delegation of technical ADRs”). It records how PLAN Sections 7.7 (the plugins’ row of the distributions table, the “wheel vs parity build” row), 7.8 and 8.8 (wheels.yml, release.yml) are implemented in M6a, step “ci-release”, and where the implementation refines the plan’s sketch. It changes no rule the author approved: the gates and their measurement protocol, the parity rules, the licence and the licence expression, the names, the publishing steps and the environments (the author’s and the orchestrator’s: testpypi / pypi for dyng, testpypi-cu12 / pypi-cu12 and testpypi-cu13 / pypi-cu13 for the plugins) are taken as given, and nothing is published by it.

  • Date: 2026-10-06

  • Deciders: the AI assistant, on the author’s behalf (M6a, step “ci-release”)

Context#

ADR 0030 builds the plugin wheels dyng-cu12 and dyng-cu13 from the core sdist with a rendered pyproject.toml; ADR 0031 lets dyng choose and fall back. Locally, ci/plugin_wheels.sh builds and tests them on the development machine. PLAN 7.7 asks for the plugins to be built by cibuildwheel in the manylinux_2_28 image “with the CUDA toolkit installed in before-all from NVIDIA’s RHEL8 repository”, PLAN 7.8 for the latest 12.x and the latest 13.x, PLAN 8.8 for wheels.yml to build and install-test them and for release.yml to publish them through Trusted Publishing, with a 90 MB size check. GitHub’s hosted runners have no GPU and no NVIDIA driver. Release 0.2.0 is the first with the plugins (PLAN Appendix F); the projects dyng-cu12 and dyng-cu13 have pending trusted publishers bound to release.yml and to their own environments (PyPI refuses two identical pending publishers, so they cannot share testpypi / pypi with dyng).

Decision#

  1. wheels.yml builds the plugins from the sdist of the same run (jobs plugin, matrix cu12 / cu13, and plugin-install-test), next to the unchanged jobs of the CPU wheel. The job unpacks the core sdist, and cibuildwheel builds it with the configuration file ci/cibuildwheel-plugin.toml (the CPU wheel keeps [tool.cibuildwheel] of pyproject.toml): build = cp312-manylinux_x86_64 (one abi3 wheel), the manylinux_2_28 image that cibuildwheel pins, before-all = ci/cibw_plugin.sh, the plugin passed in as DYNG_PLUGIN (environment-pass), CUDACXX=/usr/local/cuda/bin/nvcc, four parallel compile jobs (the runners have four cores), the module linked with -Wl,--exclude-libs,ALL (as in the local build: neither libdyng’s nor the static CUDA runtime’s symbols are exported), and the repair of ADR 0030 item 6 (auditwheel repair --plat manylinux_2_28_x86_64 --exclude libcuda.so.1 --exclude libcuda.so) in ci/cibw_plugin_repair.sh, which then checks the module’s code objects with the toolkit’s cuobjdump (ci/wheel_check.py --code-objects: exactly the SASS of the toolkit’s release list and the PTX of its last entry, PLAN 7.7; the lists in ci/wheel_check.py are tied to cmake/cuda_architectures.cmake by its self-test). twine check --strict and ci/wheel_check.py (the 90 MB budget, tags, contents, entry point, Requires-Dist: dyng==<version>, licence files, no libcuda / libcudart needed or bundled, an export list of PyInit__core and std / nanobind / type_info symbols only) run on each wheel; the artifact is wheel-cu<N>-manylinux_2_28_x86_64. Pull requests that touch what the distributions are built from build the plugins too: the packaging, the library (cpp/**, added after the M6a review: the published plugin configuration, nvcc with the image’s gcc-toolset, the static runtime and the module in plugin mode, is built by no other workflow, so a library change that breaks it would otherwise first fail at the release tag) and the licence files the wheels ship (they are not required checks); workflow_call and workflow_dispatch take an input plugins (default true).

  2. The toolkits are pinned file by file. ci/cuda_toolkits.toml, generated by ci/cuda_toolkit.py lock --plugin cu<N> --release X.Y from the metadata of NVIDIA’s RHEL 8 repository (repodata/primary.xml), pins for each plugin the toolkit release and every RPM of the minimal set the build needs, with its SHA-256 and size: cuda-nvcc, cuda-cudart-devel (with libcudart_static.a; it pulls CCCL), cuda-nvtx (NVTX 3, the profiler’s ranges), cuda-documentation (it carries EULA.txt, which the wheel ships, ADR 0030 item 9), cuda-cuobjdump (the check of the code objects) and what they require from NVIDIA’s repository; what they require from outside it (gcc-c++, /sbin/ldconfig, /bin/sh) is listed as external and comes from the image’s repositories. Pinned now: CUDA 12.9 (nvcc 12.9.86, 12 files, 183 MB) for cu12 and CUDA 13.4 (nvcc 13.4.92, 14 files, 140 MB) for cu13, the latest of each major in the repository on 2026-10-06 (PLAN 7.8). The runner downloads the files on the host (ci/cuda_toolkit.py download: every digest checked, a SHA256SUMS written) into .cuda-rpms/, which cibuildwheel copies into the container with the project; before-all checks the digests again (sha256sum --check and ci/cuda_toolkit.py verify) and installs exactly those files with dnf. The signatures are checked too (added after the M6a review): the SHA-256 pins come from the repository’s metadata over HTTPS and prove only that CI installs the bytes that were locked, while installing from NVIDIA’s repository, as the plan has it, would have checked NVIDIA’s signatures (gpgcheck=1). So download also fetches the repository key D42D0685.pub, accepts it only with the fingerprint pinned in ci/cuda_toolkit.py (610C7B14E068A878070DA4E99CD0A493D42D0685), and checks with gpg that every RPM’s header carries a valid signature by it and that the payload matches the digest the signed header records; before-all repeats that check (verify --signatures), imports the key into the container’s rpm database, requires rpm -K to report the signatures OK, and installs with dnf --setopt=localpkg_gpgcheck=1. This refines PLAN 7.7’s “installed from NVIDIA’s RHEL8 repository”: the packages are NVIDIA’s RHEL 8 packages, but the build no longer depends on what the repository serves on the day (a re-pin is a reviewed change of the lock file), and the download can be cached, which a container step cannot. Dependabot does not see the lock: re-pinning when NVIDIA ships a new toolkit is a maintainer’s step (docs/developer/wheels.md).

  3. Cache. actions/cache keeps .cuda-rpms/ between runs, keyed by the plugin and the lock file’s hash; it is not used for tag runs (a release downloads afresh), and a restored file is used only if its digest is the pinned one (the cache cannot change what is built). The build itself is not cached (no ccache in the container): about 10 to 20 minutes per plugin on the four-core runners, from the local timing.

  4. before-all checks the toolkit before the build (ci/cibw_plugin.sh): nvcc’s release is the pinned one and of the plugin’s major, EULA.txt is present, and a kernel compiles and links with the image’s host compiler (gcc-toolset 14 in cibuildwheel 4.2.1’s manylinux_2_28 image, 2026.09.05-1) and the static runtime, so an unsupported host compiler fails in one line rather than in the middle of the build. It then renders the plugin’s tree with ci/plugin_pyproject.py --cuda-root /usr/local/cuda (a link to /usr/local/cuda-X.Y).

  5. No GPU in CI: a fallback smoke test and the whole suite. plugin-install-test (plugin x Python 3.12 / 3.13) installs the CPU wheel and the plugin wheel of the run into a fresh venv (--no-index: dyng==<version> comes from the run’s core wheel, which is not on PyPI yet; NumPy and the test tools from PyPI), checks that the runner has no driver, then runs ci/plugin_smoke.py --expect fallback --reason "no CUDA driver": exactly one dyng.BackendWarning, dyng._core active, the plugin reported unusable with that reason, the plugin’s own module importing without a driver and reporting its build (plugin, toolkit, static runtime, architectures: the release list of its toolkit, in both modes), Resources.cuda() refused, and sssp / cycle_count / mosp equal on the OpenMP and sequential backends; then the whole pytest suite (its GPU tests skip). The same script with --expect cuda is the GPU smoke test of ci/plugin_wheels.sh (which now also runs the CI smoke test, with the driver hidden by ci/without_cuda_driver.sh: a user and mount namespace in which libcuda.so.1 is an empty file). The GPU tests of the plugins stay on the author’s machine (ci/plugin_wheels.sh, ci/gpu_local.sh’s step plugin), as PLAN 8.8 has it for every GPU test.

  6. release.yml publishes each distribution through its own environments. select names the distributions of the tag (ci/wheel_check.py --release-distributions, a JSON list of name, file prefix and environment suffix): dyng alone for the name reservation (v0.0.x) and for v0.1.x, and from 0.2.0, release candidates included (PLUGINS_SINCE = (0, 2)), also dyng-cu12 and dyng-cu13; it calls wheels.yml with plugins set accordingly. collect checks the files again and that they are exactly the named distributions of the version (--release-set: the core’s sdist and wheel, one wheel per plugin, no plugin sdist), and sorts them into one directory per distribution of the artifact dist (--split). Each distribution is uploaded from its directory in its own environment, the plugins before dyng on each index: publish-testpypi-plugins (a matrix over the plugins, environments testpypi-cu12, testpypi-cu13), then publish-testpypi (dyng, testpypi) only if every plugin upload succeeded; for final versions only and only after dyng reached TestPyPI, publish-pypi-plugins (pypi-cu12, pypi-cu13), then publish-pypi (dyng, pypi) only if both plugins reached PyPI. Each pypi* environment needs the author’s approval: three approvals for a final release with the plugins (the two plugins in one “Review deployments” dialog, then dyng). The order matters because dyng’s extras pin the plugins exactly: a dyng on the index without its plugin would make pip install "dyng[cu13]" backtrack to an older dyng without the extra and install the CPU package with only a warning, and a failed or rejected plugin upload (likely enough on the first upload through a new pending publisher) would leave it so. With the plugins first, such a failure leaves dyng unpublished instead, and a plugin alone is harmless (it requires dyng==<version>). docs/developer/release.md says how to recover from a failed leg. The select step’s script is tested for every kind of tag by ci/tests/test_release_select.py. For v0.0.x and v0.1.x nothing changes but the artifact’s layout (dist/dyng/): the same files go to the same environments (the plugin jobs are skipped). (Revised after the M6a review: the first version published the three distributions as one unordered matrix.)

  7. The informational “wheel vs parity build” row (PLAN 7.7) is measured by parity/wheel_vs_parity.py: the same Python driver, in two interpreters, one with the core and plugin wheels, one importing dyng from an overlay whose module is the parity-cuda preset’s library (shared libdyng.so, -O3 -lineinfo, sm_86, shared CUDA runtime) with the bindings; the workload of the gates (sssp: roadNet-CA, K = 3, the suite’s 50K unsafe batch through dyng.update as dyng-compat-mosp does, the stage sssp.enact_fused per objective; cycle_count: DD, k = 4, the stage cycle_count.count, and an update of 25K + 25K changes drawn by the script); the exclusive perf lock, GPU 0, clocks locked (boost, the ClockLock of parity/perf_ab.py), alternating rounds, medians. It is recorded in parity/results/M6a-wheel-vs-parity.json and in the M6a retrospective, never gated.

Consequences#

  • One run of wheels.yml builds and tests the sdist, the CPU wheel and both plugins from one source archive; a release publishes them together, each under its own trusted publisher.

  • The CI toolkits are reproducible and reviewable; a new CUDA toolkit needs a re-pin (and the cu12 / cu13 wheels follow PLAN 7.8’s “latest 12.x / 13.x” only through it). The locally built wheels of ci/plugin_wheels.sh differ from the CI wheels in the host compiler (conda’s GCC 12.4 with a glibc 2.28 sysroot, libstdc++ and libgcc linked statically, against the image’s gcc-toolset 14) and, unless DYNG_CUDA<N>_ROOT points at the pinned toolkit unpacked by ci/cuda_toolkit.py extract, in the toolkit (/usr/local/cuda-13.1 and conda-forge’s 12.9 on the development machine); docs/developer/wheels.md lists both.

  • The new environments testpypi-cu12, pypi-cu12, testpypi-cu13, pypi-cu13 need the protection rules of pypi / testpypi (required reviewer on pypi-*, tags v* only, no administrator bypass): an account-level setting of the author, listed in docs/developer/repository_settings.md section 11.

  • No new third-party action: actions/cache (by GitHub) is already used and SHA-pinned.

  • The CI build itself was not run on GitHub in this step (no push): the configuration was checked by actionlint, zizmor, ci/github_meta_check.py --verify-pins and cibuildwheel’s own option parsing, the pinned toolkits were unpacked and used for local builds of both plugins, once with the usual GCC 12.4 toolchain and once with conda-forge’s GCC 14 against the glibc 2.28 sysroot (the image’s compiler major), all of which passed the GPU tests on GPU 1 and the fallback smoke test with the driver hidden. The first pull request run is the first real test of the container part (dnf, the image’s gcc-toolset, cibuildwheel’s isolated build).