Nixl_ep ci: adding ep tests to dlcluster by lishapira · Pull Request #1557 · ai-dynamo/nixl

lishapira · 2026-04-20T14:26:40Z

Integrates nixl_ep elastic tests into the existing nixl-ci-dl-gpu Jenkins pipeline, running on GB300 NVL72 nodes. The tests exercise the elastic scale-up/scale-down functionality of nixl_ep for both NVLink and RDMA transports.

Changes:

CI: Added "Run DL EP elastic tests" step to test-dl-matrix.yaml; builds PR image with BUILD_NIXL_EP=true
test_ep.sh: runs elastic.py with plans: no_expansion.json and expansion_fault_contraction.json on 4 GPUs (NVLink + RDMA variants); gracefully skips on UCX v1.21.x.
Build: Added ucx_gpu_device_api_available guard to skip build nixl_ep when UCX GPU Device API is unavailable; disabled -rdc to fix nvlink register count errors; added DOCA GPUNetIO header installation in build.sh.
elastic.py: Added silent-failure assertions (unexpected rank crashes, SIGTERM count mismatch); fixed log message wording; added get_total_killed_ranks() helper (from plan.py); validation behind --validate-plan flag.

copy-pr-bot · 2026-04-20T14:26:44Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

github-actions · 2026-04-20T14:26:50Z

👋 Hi lishapira! Thank you for contributing to ai-dynamo/nixl.

Your PR reviewers will review your contribution then trigger the CI to test your changes.

🚀

itayalroy · 2026-04-20T14:27:19Z

/ok to test 42356e7

itayalroy · 2026-04-20T14:27:24Z

/build

itayalroy · 2026-04-21T12:00:18Z

/ok to test 1220c57

itayalroy · 2026-04-21T12:00:25Z

/build

itayalroy · 2026-04-23T09:32:02Z

/ok to test bfa3677

itayalroy · 2026-04-23T09:32:06Z

/build

itayalroy · 2026-04-27T17:02:36Z

/ok to test 206b36e

itayalroy · 2026-04-27T17:02:43Z

/build

itayalroy · 2026-04-28T11:48:00Z

/ok to test 015c3b9

itayalroy · 2026-04-28T11:48:04Z

/build

itayalroy · 2026-04-28T13:32:24Z

/ok to test 67529f6

itayalroy · 2026-04-28T13:32:29Z

/build

itayalroy · 2026-04-28T23:53:43Z

/ok to test e78e5ba

itayalroy · 2026-04-28T23:53:52Z

/build

- Use 4 processes instead of 8 for the elastic EP CI test. - Assert that rank failures only occur for ranks marked for kill in the plan, catching unexpected crashes during any phase. - Verify the SIGTERM exit count matches the plan's expected kills, catching cases where the fault-tolerance kill mechanism fails. - Add get_total_killed_ranks() helper to Plan class. Made-with: Cursor

Last phase now uses [0, 3] instead of [0, 1], introducing a rank-index gap (rank 1 and 2 absent) to exercise sparse rank handling. Made-with: Cursor

Made-with: Cursor

…flag Address review comment: elastic.py now exposes an optional --validate-plan flag that enables plan-specific assertions (unexpected failure rejection and SIGTERM count verification). test_python.sh passes the flag for its specific plan. Made-with: Cursor

Made-with: Cursor

The global meson.build sets -rdc=true for all CUDA targets, which causes nvlink to enforce register limits across call boundaries at link time. nixl_ep kernels use --register-usage-level=10 and call nixlPut() which uses 215 registers, exceeding the nvlink limit and failing at link time. Adding -rdc=false overrides the global setting for nixl_ep only, so nixlPut gets inlined at compile time instead of being linked separately by nvlink. This matches the standalone setup.py build behavior. Made-with: Cursor

Install DOCA GPUNetIO dev packages when PRE_INSTALLED_ENV skips the full apt bootstrap but BUILD_NIXL_EP=true requires them. UCX device headers include doca_gpunetio_dev_verbs_qp.cuh which fails compilation without the dev package installed. Unset UCX_NET_DEVICES in elastic test subshell so UCX auto-selects a GPU-capable transport. When set by the CI environment, UCX is restricted to a device without GPU peer memory support, causing the RDMA path to fail with "no lane found" errors. Bump CI_IMAGE_TAG to 20260421-1 (build.sh changed). Made-with: Cursor

DOCA device headers (.cuh) may be installed to a path that nvcc does not search by default, and meson may not find a pkg-config file for doca-gpunetio to add the include path automatically. Copy all .cuh files from the DOCA installation directory to ${CUDA_HOME}/include/ so nvcc can find doca_gpunetio_dev_verbs_qp.cuh, which is included transitively via UCX device headers. Made-with: Cursor

The UCX-master build activates UCX GPU Device API which triggers the full gdaki.cuh include chain requiring both .cuh and .h DOCA GPUNetIO headers. Add a wildcard search for all doca_gpunetio* files (any extension) from /usr/include and /opt. Bump CI_IMAGE_TAG to 20260421-3 to trigger Docker image rebuild.

…hout GPU API - Export UCX_VERSION in Dockerfile.gpu-test for test_ep.sh. - Fail EP elastic step when UCX_VERSION=master and BUILD_NIXL_EP=true but nixl_ep_cpp is missing; keep skipping on other UCX versions. - Skip examples/device/ep in Meson when UCX GPU Device API is unavailable. - Bump CI_IMAGE_TAG to 20260427-1 in build and test matrices.

Convert "Run DL EP elastic tests" from a raw sudo+ssh shell command to the slurmCI module format used by all other test steps. This fixes two bugs introduced after the rebase onto c83d742: 1. Wrong job ID file path: the old step read from /mnt/pvc/dl_job_id_<ver>_<build>.txt but the Allocate step now writes to ${JOB_ID_FILE_ROOT}/job_id_<ver>_<build>.txt, causing --slurm_job_id to be empty. 2. SSH Permission denied: the raw sudo -u svc-nixl approach never loaded the Jenkins SSH credential (svc-nixl-ssh_key), so SSH to dlcluster.nvidia.com failed with permission denied. Using slurmCI with credentialsId injects the key automatically. Bump CI_IMAGE_TAG to 20260428-1 in all three matrix YAML files to trigger a fresh base image build that incorporates the current build.sh (DOCA GPUNetIO headers block) and the PyTorch CUDA-version alignment from main (commit 1200fe5). Made-with: Cursor

basic.json and no_expansion.json are identical; remove the duplicate and use no_expansion.json consistently in all elastic test calls. Made-with: Cursor

elastic.py imports the nixl_ep package, not nixl_ep_cpp directly. Use "import nixl_ep" so the check matches the actual runtime import path. Made-with: Cursor

Co-authored-by: Cursor <cursoragent@cursor.com>

ofirfarjun7 · 2026-05-03T12:35:24Z

/build

Revert DL matrix UCX pin to master + v1.21.x. CI_IMAGE_TAG 20260428-3.

…_TAG

rakhmets · 2026-05-04T16:29:34Z

/build

pull-request-size Bot added the size/L label Apr 20, 2026

github-actions Bot added the external-contribution label Apr 20, 2026

copy-pr-bot Bot temporarily deployed to SWX_AWS April 20, 2026 14:27 Inactive

lishapira changed the title ~~Nixl_ep ci: adding ep test to dlcluster~~ Nixl_ep ci: adding ep tests to dlcluster Apr 20, 2026

lishapira force-pushed the nixl_ep_ci_adding_ep_to_dlcluster branch from 42356e7 to 1220c57 Compare April 21, 2026 11:58

copy-pr-bot Bot had a problem deploying to SWX_AWS April 21, 2026 12:00 Failure

lishapira force-pushed the nixl_ep_ci_adding_ep_to_dlcluster branch 2 times, most recently from 2f5fd3c to bfa3677 Compare April 23, 2026 07:28

copy-pr-bot Bot temporarily deployed to SWX_AWS April 23, 2026 09:32 Inactive

lishapira force-pushed the nixl_ep_ci_adding_ep_to_dlcluster branch 2 times, most recently from 3edde0e to 206b36e Compare April 27, 2026 15:56

copy-pr-bot Bot temporarily deployed to SWX_AWS April 27, 2026 17:02 Inactive

lishapira force-pushed the nixl_ep_ci_adding_ep_to_dlcluster branch from 206b36e to 3760dbc Compare April 28, 2026 09:24

copy-pr-bot Bot temporarily deployed to SWX_AWS April 28, 2026 11:48 Inactive

copy-pr-bot Bot temporarily deployed to SWX_AWS April 28, 2026 13:32 Inactive

lishapira force-pushed the nixl_ep_ci_adding_ep_to_dlcluster branch from 67529f6 to e78e5ba Compare April 28, 2026 13:42

copy-pr-bot Bot temporarily deployed to SWX_AWS April 28, 2026 13:43 Inactive

lishapira and others added 17 commits May 3, 2026 03:47

nixl_ep: Add elastic test to CI pipeline

93f8415

nixl_ep: update plan for CI test

f0efde2

Last phase now uses [0, 3] instead of [0, 1], introducing a rank-index gap (rank 1 and 2 absent) to exercise sparse rank handling. Made-with: Cursor

nixl_ep: scope PYTHONPATH to elastic CI test invocation

6f8e4e1

Made-with: Cursor

nixl_ep ci: add basic plan and NVLink/RDMA test variants for each plan

73e7caa

Made-with: Cursor

nixl_ep_ci: run elastic EP tests from DL cluster job

2ea1be6

nixl_ep ci: replace basic.json with no_expansion.json in test_ep.sh

e675eda

basic.json and no_expansion.json are identical; remove the duplicate and use no_expansion.json consistently in all elastic test calls. Made-with: Cursor

nixl_ep ci: switch test_ep.sh to /bin/bash

d32692e

nixl_ep ci: fix nixl_ep availability check in test_ep.sh

8e6fd27

elastic.py imports the nixl_ep package, not nixl_ep_cpp directly. Use "import nixl_ep" so the check matches the actual runtime import path. Made-with: Cursor

nixl_ep ci: drop --disable-ll-nvlink runs from test_ep.sh

b8a4fd4

Co-authored-by: Cursor <cursoragent@cursor.com>

lishapira force-pushed the nixl_ep_ci_adding_ep_to_dlcluster branch from a29db38 to b8a4fd4 Compare May 3, 2026 10:50

lishapira force-pushed the nixl_ep_ci_adding_ep_to_dlcluster branch from 02fb86f to c8d289c Compare May 4, 2026 08:41

nixl_ep ci: restore EP RDMA tests, trim DOCA comments; bump CI_IMAGE_TAG

a0dbb3c

Revert DL matrix UCX pin to master + v1.21.x. CI_IMAGE_TAG 20260428-3.

lishapira force-pushed the nixl_ep_ci_adding_ep_to_dlcluster branch from c8d289c to a0dbb3c Compare May 4, 2026 11:11

nixl_ep ci: pin DL UCX to 6110272d, fork clone + fetch; bump CI_IMAGE…

a193240

…_TAG

lishapira force-pushed the nixl_ep_ci_adding_ep_to_dlcluster branch from d1e842b to 79c3e36 Compare May 4, 2026 16:08

nixl_ep ci: print ucx_info -v at start of test_ep.sh

dfc1312

lishapira force-pushed the nixl_ep_ci_adding_ep_to_dlcluster branch from 79c3e36 to dfc1312 Compare May 4, 2026 16:12

Conversation

lishapira commented Apr 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

copy-pr-bot Bot commented Apr 20, 2026

Uh oh!

github-actions Bot commented Apr 20, 2026

Uh oh!

itayalroy commented Apr 20, 2026

Uh oh!

itayalroy commented Apr 20, 2026

Uh oh!

itayalroy commented Apr 21, 2026

Uh oh!

itayalroy commented Apr 21, 2026

Uh oh!

itayalroy commented Apr 23, 2026

Uh oh!

itayalroy commented Apr 23, 2026

Uh oh!

itayalroy commented Apr 27, 2026

Uh oh!

itayalroy commented Apr 27, 2026

Uh oh!

itayalroy commented Apr 28, 2026

Uh oh!

itayalroy commented Apr 28, 2026

Uh oh!

itayalroy commented Apr 28, 2026

Uh oh!

itayalroy commented Apr 28, 2026

Uh oh!

itayalroy commented Apr 28, 2026

Uh oh!

itayalroy commented Apr 28, 2026

Uh oh!

ofirfarjun7 commented May 3, 2026

Uh oh!

rakhmets commented May 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

lishapira commented Apr 20, 2026 •

edited

Loading