Retrospective: M5 (the Python package, the CLI and the 0.1 documentation)

Status: complete (2026-09-30) on the branch m5-python (from main at a826ed0, the merge of pull request #3 with M3), pending the orchestrator’s pull request into main. Nothing was published: no tag, no upload to TestPyPI or PyPI, no GitHub Release. M5 was done in three steps, each by one implementer, followed by a review round that fixed the confirmed findings of the independent reviewers (section “Review round”); this page was written in step 3 from the steps’ records (ADRs 0011 and 0025, docs/developer/python_gaps.md, docs/developer/wheels.md), updated in the review round, and ends with the readiness list of release 0.1.0.

What was built

Step	Commits	What
1. bindings	7231a00..ac62269	DYNG_BUILD_PYTHON and the extension module dyng._core (nanobind 3.1.0, stable ABI, libdyng linked statically, NB_DOMAIN dyng_cpu); the root pyproject.toml (scikit-build-core, abi3 cp312, the version from VERSION); the typed layer python/dyng (Resources, Graph with dtype dispatch, EdgeBatch, dyng.sssp, dyng.cycle_count, dyng.update, the exception hierarchy, dyng.Array, dyng.io, dyng.generators.legacy, dyng.testing, profile(), citation(), show_config(), algorithms()); the committed stubs and regen.py --stubs --check; the pytest suite with the Hypothesis profile and the Python-level parity subset; ci/python.sh; ADR 0011 and the Python gaps
2. wheels-cli	d14313d..1a11913	the dyng command line (sssp, cycle_count, prep, convert, generate, config; flags generated from the option fields; the originals’ formats; 51 tests against the originals’ fixtures); [tool.cibuildwheel], ci/wheel_check.py (the 90 MB budget), ci/wheels.sh (the local manylinux_2_28 build with a glibc 2.28 toolchain), wheels.yml, python.yml, release.yml for tags v0.1.0 and later; ADR 0025, the CLI reference, the wheels page
3. docs-finish	f2e27c3..(this step’s last commit)	the Python API reference (sphinx-autoapi), getting started in Python, the README quickstarts run by pytest and CTest, examples/python, the Python and CLI sections of the algorithm pages, “Port research code”, the history pages, api-check.yml (griffe), the release process page, the repository state of 2026-09-30 (DCO required, signing), the CHANGELOG draft of 0.1.0, this retrospective and the verification from a fresh clone

Acceptance criteria

#	Criterion	Evidence	Status
1	Root pyproject.toml (scikit-build-core + nanobind, name dyng, dynamic version, Python >= 3.12, abi3 cp312, DYNG_BUILD_PYTHON=ON, CUDA off); pip install ., python -m build, the editable install; tools/name_reservation still builds but release.yml builds the real package from v0.1.0	step 1 (pip install ., python -m build, the editable install in the dev env and in clean venvs); step 2 (release.yml’s select job; the name-reservation job still runs in lint.yml)	met
2	The Python API of PLAN 5.4 for 0.1	step 1 (ADR 0011; what is not bound is listed in python_gaps.md); the review round (the .dgt batch files and dyng.io.read_batches(), which the first version had moved to 0.2 without an ADR; use_cpu_only() before first use; exports that survive updates)	met, with the gaps of python_gaps.md (.dgb binary batches, dyng.interop, memory resources, CUDA)
3	pytest: API, DLPack/NumPy, exceptions, stale results, Hypothesis, Python-level parity; green in the dev env and in a clean venv with only the wheel	steps 1 and 2; step 3 adds the snippet and example tests; the review round the growth, copy, pickle, thread and scalar tests (section “Verification”)	met
4	manylinux_2_28 abi3 wheel with libgomp and the sdist, locally and in wheels.yml; installable on 3.12 and 3.13; python.yml; release.yml for v0.1.0 and later; nothing published	step 2 (ci/wheels.sh: 1.75 MB wheel, 0.77 MB sdist, the full suite on 3.12.14 and 3.13.15); re-run after the review round (section “Verification”)	met locally; the hosted workflows have not run yet (readiness item 4)
5	The dyng CLI (PLAN 5.6)	step 2 (ADR 0025: a console script of the Python package instead of the C++ tools/cli)	met
6	Documentation for 0.1; README snippets executed; docs with -W; the repository state recorded	this step (below)	met
7	A fresh clone passes ci/check.sh --parity, ci/gpu_local.sh, ci/docs.sh and the Python checks; the portability pre-checks; CHANGELOG; this retrospective with the readiness list	section “Verification”	met

Review round

Independent reviewers (lenses: Python API, packaging, CLI and docs, regression) reported 36 findings that a skeptic confirmed at 1035309; three were the same (the empty “Verification” section). Every one is fixed on the branch; none was deferred.

#	Finding (severity)	Resolution	Commit
1	Zero-copy exports (np.from_dlpack, np.asarray, to_numpy(copy=False)) read freed memory after an update that grows the vertex set (high)	the result state lives in a shared_ptr (result_holder); every export holds it; an update copies the state while an export is alive (copy-on-write). Tests grow a 20,000-vertex path by 30,000 vertices under four export kinds, and a reader thread races 40 growing updates	ab5c6e4
2	copy.copy(result) bypasses the staleness check (high)	the generation counter is in the native holder; Result.__copy__ / __deepcopy__ clone	ab5c6e4
3	EdgeBatch caches converted ids but reads same-dtype arrays live (high)	no cache: the arrays are converted at every use; tests refill the arrays between two applies for int32/int64 ids and weights	a3e155e
4	.dgt and dyng.io.read_batches missing (medium)	implemented in C++ (io::read_batches / write_batches) and Python; the CLI reads one-batch .dgt files; PLAN 5.5’s loop is tested	399bf64
5	pickling / copying: obscure errors, a used batch unpicklable (medium)	ADR 0011 item 13	ef52b03
6	use_cpu_only() cannot work; _core always loaded (medium)	the native module is chosen on first use (ADR 0011 item 14); a fake plugin entry point tests both paths	e7c4c2b
7	scalars coerced with int() / bool() (medium)	as_int / as_bool (operator.index semantics) everywhere user input meets a native call	54d89a9
8	profile() / copy_policy race with running calls (medium)	both under while_idle; the native-call lock prefers writers (with glibc’s std::shared_mutex the new thread test starved)	d4a56ed
9	legacy DLPack exports writable (low)	unversioned exports are copies; to_torch() documented as not writable	f5fc278
10	buffer-protocol int64 inputs select int32 ids (low)	the buffer format is the declared dtype	6b83977
11	no mypy in CI; 90 errors (low)	mypy --strict clean, in ci/python.sh and python.yml; overloads for dyng.update	6f3f556
12	wrong ADR cross-references (low)	ADR 0011 items 3 and 5	ab5c6e4, 6b83977
13	option defaults not tested against C++ (low)	a parametrized test per dataclass	e0fe403
14	the wheel ships nanobind, robin-map and the GCC runtime without their licences (high)	THIRD_PARTY_LICENSES.txt in license-files, required by ci/wheel_check.py; the License-Expression question goes to the author (readiness item 15)	513cbb5
15	release.md steps 5/7/9 contradict release.yml (medium)	VERSION is X.Y.ZrcN for a candidate; a separate row for the final bump	513cbb5
16	the “Verification” section was a placeholder (medium; three reviewers)	filled in below	this page
17	the published sdist is not what the wheel is built from (low)	wheels.yml builds the wheel from the sdist artifact	513cbb5
18	deprecated tool.scikit-build.metadata, unbounded requirement (low)	[[tool.dynamic-metadata]], scikit-build-core>=1.1.0,<2; no warning in the build	513cbb5
19	the sdist ships the CC-BY-SA-4.0 Code of Conduct and repository files (low)	excluded; wheel_check rejects them	513cbb5
20	extras differ from PLAN 7.7 without a record (low)	ADR 0011 item 15	513cbb5
21	the local wheel bundles an unpinned GCC 16 libgomp (low)	libgomp=12.* (and libgcc, libstdcxx) in ci/wheel-toolchain.yml; the toolchain environment of this machine was updated to 12.4.0	513cbb5
22	tag checks compare the raw VERSION with normalised names (low)	VERSION must be canonical PEP 440 (ci/wheel_check.py --version-info), the pre-release flag comes from the same parse	513cbb5
23	dyng cycle_count update without --max-length hangs (medium)	a usage error as in the original; fixture c11 asserted; ADR 0025 item 4 corrected	43924a8
24	the “add an algorithm” guide’s step 7 (medium)	rewritten for the M5 layout	f0d0ee7
25	a getting-started link to a non-existent CLI example (low)	reworded	c653b68
26	the CUDA message points to a missing section (low)	“use CUDA from C++”, the install page’s Python section	c653b68
27	CLI errors in Python/C++ terms, doubled dyng: (low)	hints with --random-weights, a readable --source check, one prefix	43924a8
28	--objective help “(default: 0)”; --validate-inputs on compute (low)	“every weight column”; the flag only on update	43924a8
29	the cycle-enum flag table misses renamed flags and values (low)	rows added; the missing stderr line is noted	43924a8
30	the mapping tables do not name the CLI (low)	added	f0d0ee7
31	MOSP-OpenMP history: -O3 baseline, packed-format fix, M-f (low)	corrected	f0d0ee7
32	ci/python.sh re-points the shared dyng-dev environment (medium)	a throwaway venv when the environment’s editable dyng is another checkout	9a93d8b
33	DYNG_BUILD_DOCS still a no-op (low)	the target docs	7f37655
34	the bindings are never compiled by Clang (low)	a Clang 18 entry in python.yml; the local pre-check skips nanobind’s sources	0405585

The CHANGELOG has the “Fixed (M5 review)” and “Changed (M5 review)” entries; the Python reference (docs/api/python/index.md) states the new array, copy and thread rules (31e1563).

Step 3: documentation and close-out

Done

• Python API reference (PLAN 9.2): sphinx-autoapi 3.8.1 reads python/dyng and the committed stubs statically (no compiled module, so the docs job, Read the Docs and CPU-only machines need no build). The reference is the public API as users import it: the dyng package page with every name of dyng.__all__, and one page each for dyng.sssp, dyng.cycle_count, dyng.io, dyng.generators and dyng.testing. The implementation modules whose names the package re-exports (dyng.graph, dyng.array, …) get no page of their own, and a missing-reference hook resolves the signatures’ dyng.graph.Graph to the documented dyng.Graph. The docstrings needed two fixes for the reference: the #: field comments of the dataclasses (which autoapi does not read) became “Attributes” sections, and nine property docstrings that started with “text: …” (which napoleon reads as a type) were reworded. docs/api/python/index.md explains the package-wide rules (resources, dtype dispatch, arrays, stale results, the exception table, threads, profiling).

• Getting started: the index page shows both ten-line quickstarts (included from the README, so they are the tested ones); install.md has the Python section; “A first update in Python” (examples/python/first_update.py, the twin of the C++ page, and a dyng.update() snippet with sssp and cycle_count on one graph).

• README quickstarts executed (PLAN 9.6): marked code blocks (<!-- snippet: name -->, an optional output block) in the READMEs and the pages. python/tests/test_doc_snippets.py runs every Python snippet (6 today: the two READMEs, the first update, both algorithm pages) and compares its output; the CTest example.readme_quickstart extracts the C++ quickstart at configure time, compiles it with the examples’ warnings and compares its output; ci/doc_snippets.py --check (in ci/docs.sh, tested by ci/tests/test_doc_snippets.py) keeps the quickstarts present and within ten lines.

• examples/python (first_update.py, sssp_update.py, cycle_count_update.py, the twins of the C++ examples), tested against the originals’ outputs by python/tests/test_examples.py.

• Algorithm pages: a runnable Python snippet with its output and the command lines on the originals’ formats in the API sections of sssp and cycle_count; the “Python binding arrives in M5” notes are gone. The update-model page shows the profiler stages from Python.

• How-to “Port research code” (PLAN 6.3), written from what the sssp and cycle_count ports did, with the harness’s real commands.

• History pages for MOSP-OpenMP, MOSP-CUDA and CycleEnumeration-GPU (PLAN 9.2): provenance, the pinned commit and the paper’s snapshot, the ported files, the fixes of the original before the port, what differs and what stays behind.

• api-check.yml (a PLAN 11.2 M5 workflow, left open by step 2) with ci/api_check.sh: griffe 2.3.0 against the base branch; a breaking change needs the api-change label and a CHANGELOG entry (PLAN 5.9). The api step of ci/check.sh runs it locally.

• Release process (docs/developer/release.md, PLAN 10.3), referenced by wheels.md and by an error message of release.yml since step 2.

• Repository state of 2026-09-30 (pending_author_facts.md): the approvals-log row in GOVERNANCE.md and steps 7 and 9 of repository_settings.md (DCO required, 18 checks, the signing key; the pyyaml pin of the harness job).

• The roadmap’s “Where we are”, the landing page and the README say where 0.1.0 stands; the CHANGELOG has the draft summary of 0.1.0 at the top of Unreleased.

Deviations from the plan (pragmatic choices, same intent)

Plan	What was done	Why
PLAN 9.2: sphinx-autoapi “over the nanobind stubs (.pyi)”	autoapi over the typed layer python/dyng/*.py and the stubs	the public API is the typed layer (ADR 0011); the stubs describe only the private dyng._core. Both are read statically, so the intent (no compilation for the docs) holds
PLAN 9.6: the README quickstarts are run “by a CI step (ci/docs.sh)”	the Python snippets run in pytest (python.yml, ci/python.sh), the C++ one in CTest (cpu.yml, ci/check.sh); ci/docs.sh runs the structural check	the docs job configures but builds nothing (Doxygen, Sphinx), and running the snippets needs the compiled library; the jobs that build it run them, so they still cannot drift
PLAN 9.6: “10 lines of C++”	ten lines in the body of main() (with the two includes and the braces, 15 lines); ten non-blank lines of Python	a complete, compilable program; ci/doc_snippets.py counts it this way
PLAN 9.2 tutorials “Dynamic SSSP in Python in 10 minutes” and “Cycle counting on a changing graph” (0.1)	moved to 0.1.x; “A first update in Python” and the algorithm pages’ Python sections cover the steps	not in the M5 criteria; recorded in docs/tutorials/index.md and the roadmap
PLAN 10.3 step 7: “GitHub Release from the CHANGELOG section” by release.yml	a manual step of the author (gh release create), documented in release.md	a job with contents: write on tags is a publishing change; left to the author with the tag
PLAN 5.9: api-check.yml checks against “the last tag”	against the pull request’s base branch	the only tag, v0.0.1, is the name reservation without an API; against the base branch every pull request is checked from now on. From 0.1.0 a release branch can compare with the tag (ci/api_check.sh v0.1.0)
PLAN 5.7: .dgt holds vertex (+v, -v) and hypergraph (+h, -h, +i, -i) operations	the 0.1 reader rejects them with an io_error naming the release that brings them	edge_batch has no vertex operations before label_propagation (0.3) and there are no hypergraph batches before 0.2; the syntax is reserved, so files written later stay readable (review round)
PLAN 8.7’s tooling table: mypy through pre-commit	mypy --strict in ci/python.sh and python.yml	the check needs NumPy and the installed typed layer, which pre-commit’s isolated environments do not have (review round)
PLAN 6.3 step 10: maturity stable at the end of a port	sssp and cycle_count stay experimental	step 10 also asks for the benchmark-suite record (benchmarks/, PLAN 8.5), which does not exist yet; the change is left for the release decision (readiness item 12)

Notes

• autoapi_keep_files is off: the generated pages exist only during the build (and are git-ignored). The source-read hook gives them the Python default domain, because the site’s primary domain is C++.

• The pure docstring changes of the typed layer changed no signature: ci/api_check.sh against 1a11913 (the end of step 2) reports no breaking change.

Verification

A fresh clone of the branch at 524e96a (all review fixes; the later commits change only this page and docs/api/python/index.md, whose site build ci/docs.sh passed on the final commit), made with git clone /home/sskg8/Projects/dyng and git checkout m5-python in $DYNG_SCRATCH/review/m5-fix-verify, on 2026-09-30, by the script $DYNG_SCRATCH/runs/m5-fix-verify/run.sh (logs next to it). Heavy steps under the shared lock, niced; GPU tests on GPU 1.

Check	Command	Result
CPU gate with parity	ci/check.sh --parity	all checks passed (26 min): clang-format; cpu-only 592/592 and dev 605/605 CTests; clang-tidy; reuse; provenance (141 files); harness; python (the clone used the throwaway venv build/py-venv, the shared dyng-dev environment kept pointing at the main checkout: stubs current, mypy --strict clean, pytest 396 passed, 4 skipped for torch / cupy / pandas); api (origin/main has no Python package yet); scaffold (80 probe tests); docs (Doxygen coverage, API baseline 627 declarations, Sphinx -W, links); pre-commit; parity 4/4 (parity.sssp.mosp_openmp_c352151, parity.cycle_count.cycle_enum_0a976ad, CycleCountDatasets, CycleEnumDatasets)
GPU gate	ci/gpu_local.sh (dev-cuda)	all steps passed (31 min): gpu 252/252, cpu 605/605 in the CUDA build, sssp goldens on cuda 495/495, cycle_count goldens 48/48 replays (24 cases on cuda and cuda:resident), memcheck, synccheck, racecheck (0 errors), clang-tidy
Distributions	ci/wheels.sh	sdist 0.79 MB and wheel 1.50 MB (cp312-abi3-manylinux_2_28_x86_64, libgomp of conda-forge GCC 12.4.0 bundled, THIRD_PARTY_LICENSES.txt in .dist-info/licenses); twine and wheel_check ok; the wheel alone in fresh venvs: Python 3.12 and 3.13, 396 passed, 4 skipped each
Clang 18 pre-check	clang_syntax.py over a dev compile database with -DDYNG_BUILD_PYTHON=ON	174 translation units (library, tests, compat tools and the bindings), 0 failures
Fortified GCC	cmake --preset cpu-only -DCMAKE_CXX_FLAGS=-D_FORTIFY_SOURCE=3 -DDYNG_BUILD_PYTHON=ON	built, 0 warnings

Performance: the review round changed no code of a timed region. git diff 1035309..HEAD -- cpp/ adds the .dgt reader and writer (cpp/src/io/batch_text.cpp), their tests and the API baseline, and nothing else; the reviewers’ A/B at 1035309 against an a826ed0 parity build (sssp roadNet-PA 0.996-1.008, cycle_count 0.978-1.003) therefore still describes the library. The Python-side copy-on-write copies a result only while an exported view is alive; the CLI and the examples keep none across an update.

Release 0.1.0 readiness

The checklist of PLAN 10.3 (Release process) for 0.1.0, with the state on the branch m5-python. “Author” items are publishing or account-level steps, which are not delegated.

#	Item	State	Who
1	Full parity on the GPU machine	done from a fresh clone at 524e96a (see “Verification”)	done here; repeat on the release commit
2	Performance gates and the parity certificates	the gates are recorded in parity/results/M1a.md to M3.md (the last full suite on the M3 code, parity/results/M3.md section 6; M5 changed no library or CUDA code); the machine-readable release certificate benchmarks/results/0.1.0/parity.json does not exist yet	to do on the release commit (maintainer or AI assistant, on request)
3	Sanitizers and mutation checks	compute-sanitizer memcheck / synccheck / racecheck in ci/gpu_local.sh; the mutation CTests in the dev preset; the asan / tsan presets were not re-run in M5	re-run on the release commit
4	Distributions build and install	ci/wheels.sh locally at 524e96a (see “Verification”); wheels.yml has not run on GitHub yet	orchestrator: first hosted run on the M5 pull request (needs item 14)
5	VERSION 0.1.0.dev0 -> 0.1.0rc1 for the candidate, later 0.1.0 (release.md steps 5 and 9: the tag must equal VERSION, in canonical PEP 440 form); the CHANGELOG Unreleased -> [0.1.0rc1], then [0.1.0] - <date>	the CHANGELOG draft summary exists	release pull requests (maintainer), after the author’s go
6	CITATION.cff version and date-released	not yet (still the pre-release record)	the same pull request
7	Tag v0.1.0rc1 -> TestPyPI; smoke install on a clean CPU machine and on the GPU machine	not done: nothing is published in M5	author (tag); then maintainer (smoke install)
8	Tag v0.1.0 -> TestPyPI, then PyPI after the approval of the pypi environment	not done	author (tag, approve the deployment)
9	GitHub Release from the CHANGELOG section	not done	author
10	Zenodo DOI, Read the Docs	not connected (checkpoint A4, M6)	author, later (A4)
11	Announce; curate the good-first-issue backlog	not done	maintainer, after the release
12	Maturity stable for sssp and cycle_count (PLAN 6.3 step 10)	both experimental; the benchmark-suite record of PLAN 8.5 is missing	decision before the release commit (author)
13	Required checks: python.yml’s four jobs (the dev job for gcc and clang-18, the two sdist jobs) and api-check.yml’s Python API (griffe) in the main ruleset	to add after they have run on the M5 pull request (repository_settings.md, step 9)	author (account-level)
14	pypa/cibuildwheel@* on the Actions allow-list	to add before wheels.yml can run (repository_settings.md, step 5)	author (account-level)
15	Licence metadata of the wheel: it bundles nanobind (BSD-3-Clause), robin-map (MIT) and the GCC runtime (GPL-3.0-or-later WITH GCC-exception-3.1), whose texts THIRD_PARTY_LICENSES.txt now ships; License-Expression is still Apache-2.0. Should it name the bundled licences (NumPy’s style: Apache-2.0 AND BSD-3-Clause AND MIT AND GPL-3.0-or-later WITH GCC-exception-3.1)?	open: licensing is not delegated (GOVERNANCE.md)	author (decide; record in the approvals log)