feat(tests): add pytest-codeblocks for documentation code snippet CI#198
feat(tests): add pytest-codeblocks for documentation code snippet CI#198haoyu-haoyu wants to merge 4 commits intodotimplement:mainfrom
Conversation
Add automated testing infrastructure for documentation code examples using pytest-codeblocks: - Add pytest-codeblocks>=0.17.0 to dev dependencies - Add [tool.pytest.ini_options] with codeblocks marker - Add doc snippet test step to CI workflow - Mark 56 Python code blocks across 11 doc files with skip markers (these require healthchain package + external services to run) The CI step runs with continue-on-error initially so snippets can be incrementally enabled as they are made self-contained. Usage: uv run pytest --codeblocks docs/ # Test all doc snippets uv run pytest --codeblocks docs/quickstart.md # Test specific file To make a snippet testable, remove its <!-- pytest-codeblocks:skip --> marker and ensure it runs standalone (no external dependencies). Closes dotimplement#164
- Fix skip marker format: replace legacy `pytest-codeblocks:skip` with current `pytest.mark.skip` syntax (56 markers updated) - Restrict CI doc test to Python 3.12 only (pytest-codeblocks not verified for 3.13) - Narrow CI scope to quickstart + cookbook + tutorials (avoids executing shell blocks in reference docs) - Update pyproject.toml comment with correct skip syntax
- Add skip markers to 3 missed cookbook files (format_conversion, index, setup_fhir_sandboxes) — 10 Python blocks - Add skip markers to all bash/sh/shell blocks in scoped docs to prevent pytest-codeblocks from executing install/setup scripts
adamkells
left a comment
There was a problem hiding this comment.
Hey @haoyu-haoyu great work on this! Apologies for the delay in reviewing. I left a couple of minor comments. I think it would be great to have at least one example unskipped so that we can verify that the changes are working correctly.
| # Snippets can be skipped with: <!--pytest.mark.skip--> | ||
| # Entire files can be skipped with: <!--pytest-codeblocks:skipfile--> | ||
| markers = [ | ||
| "codeblocks: marks tests extracted from documentation code blocks", |
There was a problem hiding this comment.
I'm not sure if this option does anything as we don't use a codeblocks marker in any of our existing pytests. Can you confirm what this addition does?
|
|
||
| ```bash | ||
| <!--pytest.mark.skip--> | ||
| ```bash |
There was a problem hiding this comment.
There are a few instances like this where the indentation has been thrown off a little. Can you check these?
1. Unskip 3 runnable examples in docs/quickstart.md so CI actually
verifies pytest-codeblocks integration:
- Pipeline quick-inline (line ~61)
- sandbox.list_available_datasets (line ~207)
- fhir.create_condition helper (line ~264)
All three are in-memory only (no network, no filesystem deps).
Also fix a latent bug in the pipeline snippet: create_condition()
requires subject=..., and problem_list is a getter that returns a
fresh list per call — append() on it doesn't persist. Changed to
the setter: `doc.fhir.problem_list = [condition]`.
2. Remove dead [tool.pytest.ini_options] markers block from
pyproject.toml. pytest-codeblocks uses the --codeblocks CLI flag,
not a pytest marker, and no tests reference @pytest.mark.codeblocks.
3. Fix markdown code-fence indent mismatches introduced by the earlier
skip-marker insertion pass. Walked every docs/**/*.md, matched
each opener/closer pair, re-indented openers to match closer
indent. 18 fixes across 7 files (ml_model_deployment,
multi_ehr_aggregation, setup_fhir_sandboxes, and 4 tutorial
files). Restores correct rendering of snippets nested under
admonitions / numbered lists.
4. Drop `continue-on-error: true` from the codeblocks CI step.
Skipped snippets are already non-blocking by design, so the flag
only masked real failures in the now-live examples.
Verified:
uv run pytest --codeblocks docs/quickstart.md docs/cookbook/ \\
docs/tutorials/ → 3 passed, 107 skipped
uv run pytest → 746 passed, 13 skipped
uv run ruff check . → clean
|
Thanks @adamkells! Pushed fixes for all three in 1. At least one real unskipped example — unskipped 3 snippets in
All three are in-memory only (no network, no filesystem deps), so CI won't get flaky. 2. Dead 3. Indentation — there were 18 mismatched fence pairs across 7 files (all from my earlier skip-marker insertion pass — I un-indented openers but left content/closers at the original indent, breaking any block nested under admonitions or lists). Wrote a pass over Bonus — also dropped Verified locally: Happy to unskip more examples in follow-up PRs as we confirm they work end-to-end (the cookbook ones mostly need external services so they'll stay skipped for now). |
Summary
Add automated testing infrastructure for documentation code examples using
pytest-codeblocks, so code snippets stay in sync with the evolving codebase.Changes
New dependency
pytest-codeblocks>=0.17.0,<0.18added to dev dependenciesCI integration
.github/workflows/ci.ymlrunspytest --codeblocksondocs/quickstart.md,docs/cookbook/, anddocs/tutorials/continue-on-error: trueso snippets can be incrementally enabledSkip markers
<!--pytest.mark.skip-->(these require healthchain package + external services)pytest configuration
[tool.pytest.ini_options]withcodeblocksmarker inpyproject.tomlUsage
To make a snippet testable, remove its
<!--pytest.mark.skip-->marker and ensure it runs standalone.Design decisions
quickstart,cookbook, andtutorialsare tested — reference/API docs are excluded to avoid executing auto-generated content.Closes #164