fix: adding k8s taints for ephermeral jobs by theothermike · Pull Request #2420 · NVIDIA/Megatron-LM

theothermike · 2025-11-27T18:53:50Z

What does this PR do ?

⚠️ For major changes (either in lines of code or in its impact), please make sure to first share discuss a design-doc with the team.

Contribution process

flowchart LR
    A[Pre-checks] --> B[PR Tests]
    subgraph Code Review/Approval
        C1[Expert Review] --> C2[Final Review]
    end
    B --> C1
    C2 --> D[Merge]

Pre-checks

I want this PR in a versioned release and have added the appropriate Milestone (e.g., Core 0.8)
I have added relevant unit tests
I have added relevant functional tests
I have added proper typing to my code Typing guidelines
I have added relevant documentation
I have run the autoformatter.sh on my PR

Code review

The following process is enforced via the CODEOWNERS file for changes into megatron/core. For changes outside of megatron/core, it is up to the PR author whether or not to tag the Final Reviewer team.

For MRs into `main` branch

(Step 1): Add PR label `Expert Review`

(Step 2): Collect the expert reviewers reviews

Attach the Expert Review label when your PR is ready for review.
GitHub auto-assigns expert reviewers based on your changes. They will get notified and pick up your PR soon.

⚠️ Only proceed to the next step once all reviewers have approved, merge-conflict are resolved and the CI is passing.
Final Review might get declined if these requirements are not fulfilled.

(Step 3): Final Review

Add Final Review label
GitHub auto-assigns final reviewers based on your changes. They will get notified and pick up your PR soon.

(Optional Step 4): Cherry-pick into release branch

If this PR also needs to be merged into core_r* release branches, after this PR has been merged, select Cherry-pick to open a new PR into the release branch.

For MRs into `dev` branch

The proposed review process for `dev` branch is under active discussion.

MRs are mergable after one approval by either eharper@nvidia.com or zijiey@nvidia.com.

Merging your PR

Any member of core-adlr and core-nemo will be able to merge your PR.

copy-pr-bot · 2025-11-27T18:53:54Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

theothermike · 2025-11-27T18:54:31Z

/ok to test fd27b23

theothermike · 2025-11-27T19:21:06Z

/ok to test fd27b23

theothermike · 2025-11-27T20:15:25Z

/ok to test fd27b23

* [DEV] pull main Nov 25 (NVIDIA#2395) Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com> Signed-off-by: oliver könig <okoenig@nvidia.com> Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> Signed-off-by: dimapihtar <dpihtar@gmail.com> Signed-off-by: Youngeun Kwon <youngeunk@nvidia.com> Signed-off-by: Youngeun <kyeg9404@gmail.com> Signed-off-by: Maanu Grover <maanug@nvidia.com> Signed-off-by: ykarnati <ykarnati@nvidia.com> Signed-off-by: Deepak Narayanan <dnarayanan@nvidia.com> Signed-off-by: GitHub Actions <github-actions[bot]@users.noreply.github.com> Signed-off-by: Charlie Truong <chtruong@nvidia.com> Signed-off-by: Zhongbo Zhu <zhongboz@nvidia.com> Signed-off-by: Xiaowei Ren <xren@nvidia.com> Signed-off-by: Xin Yao <xiny@nvidia.com> Signed-off-by: Keshav Santhanam <ksanthanam@nvidia.com> Signed-off-by: Pablo Garay <pagaray@nvidia.com> Signed-off-by: Asha Anoosheh <aanoosheh@nvidia.com> Signed-off-by: Chen Cui <chcui@nvidia.com> Co-authored-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com> Co-authored-by: Yashaswi Karnati <144376261+yashaswikarnati@users.noreply.github.com> Co-authored-by: Jared Casper <155158+jaredcasper@users.noreply.github.com> Co-authored-by: Antoni-Joan Solergibert <asolergibert@nvidia.com> Co-authored-by: Jianbin Chang <shjwudp@gmail.com> Co-authored-by: oliver könig <okoenig@nvidia.com> Co-authored-by: Ananth Subramaniam <ansubramania@nvidia.com> Co-authored-by: Teodor-Dumitru Ene <34819528+tdene@users.noreply.github.com> Co-authored-by: Siddharth Singh <136645615+sidsingh-nvidia@users.noreply.github.com> Co-authored-by: Mcore Bot <mcore-bot@nvidia.com> Co-authored-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com> Co-authored-by: Youngeun Kwon <youngeunk@nvidia.com> Co-authored-by: Lawrence McAfee <85179052+lmcafee-nvidia@users.noreply.github.com> Co-authored-by: Maanu Grover <109391026+maanug-nv@users.noreply.github.com> Co-authored-by: Lawrence McAfee <lmcafee@nvidia.com> Co-authored-by: AJ Schmidt <ajschmidt8@users.noreply.github.com> Co-authored-by: Deepak Narayanan <2724038+deepakn94@users.noreply.github.com> Co-authored-by: helen ngo <helenn@nvidia.com> Co-authored-by: GitHub Actions <github-actions[bot]@users.noreply.github.com> Co-authored-by: Aaron Gokaslan <aaronGokaslan@gmail.com> Co-authored-by: Robert Kirby <rkirby@nvidia.com> Co-authored-by: Teodor-Dumitru Ene <tene@nvidia.com> Co-authored-by: yeyu-nvidia <yeyu@nvidia.com> Co-authored-by: Abhinav Khattar <akhattar@nvidia.com> Co-authored-by: Roger Waleffe <rwaleffe@nvidia.com> Co-authored-by: Charlie Truong <chtruong@nvidia.com> Co-authored-by: Tong Liu <liutongt1998@gmail.com> Co-authored-by: Zhongbo Zhu <42691305+zhongbozhu@users.noreply.github.com> Co-authored-by: Xiaowei Ren <xren@nvidia.com> Co-authored-by: Xin Yao <xiny@nvidia.com> Co-authored-by: Teodor-Dumitru Ene <teodord.ene@gmail.com> Co-authored-by: Zijie Yan <zijiey@nvidia.com> Co-authored-by: root <root@pool0-01101.cm.cluster> Co-authored-by: Keshav Santhanam <ksanthanam@nvidia.com> Co-authored-by: Pablo Garay <pagaray@nvidia.com> Co-authored-by: Asha Anoosheh <aanoosheh@nvidia.com> Co-authored-by: Kan Zhu <kanz@nvidia.com> Co-authored-by: Robert Kirby <rkirby@cw-dfw-cs-001-vscode-01.cm.cluster> Co-authored-by: Jorge Albericio <jalbericiola@nvidia.com> Co-authored-by: Jon Barker <19699370+jon-barker@users.noreply.github.com> Co-authored-by: Chen Cui <chcui@nvidia.com> Co-authored-by: Pablo Garay <palenq@gmail.com> Co-authored-by: Tong Liu <tongliu@nvidia.com> * adding action for checking whether PR author is nvidia employee or not for selecting ephemeral ci hosts (NVIDIA#2402) Signed-off-by: oliver könig <okoenig@nvidia.com> * fix: exit failure when PR author is external contributor removed (NVIDIA#2410) * fix: adding k8s taints for ephermeral jobs (NVIDIA#2420) * ci: Enable functional tests (NVIDIA#2419) Signed-off-by: oliver könig <okoenig@nvidia.com> * Reapply "build: Upgrade deps (NVIDIA#2289)" (NVIDIA#2408) Signed-off-by: oliver könig <okoenig@nvidia.com> * fix: use a script to do node tainting in the cicd workflow (NVIDIA#2421) * Revert "[DEV] pull main Nov 25 (NVIDIA#2395)" This reverts commit 56682f8. Signed-off-by: oliver könig <okoenig@nvidia.com> * [Dev] Support packed seq in MTP (NVIDIA#2043) Signed-off-by: Li Tao <lit@nvidia.com> Signed-off-by: lit <lit@nvidia.com> * Fix runaway Etpt in straggler detector by resetting FLOPs accumulator (NVIDIA#2128) Signed-off-by: Santosh Bhavani <santosh.bhavani@live.com> Co-authored-by: Li Ruixiao <cgruixiao@outlook.com> * [Dev] feat(MoE): Refactor cuda_graph_scope - part2 (NVIDIA#2353) Signed-off-by: Robin Zhang <robinz@nvidia.com> * [dev] DeepSeek V3.2 support (NVIDIA#2154) Signed-off-by: kunlunl <kunlunl@nvidia.com> * Revert "[Dev] feat(MoE): Refactor cuda_graph_scope - part2 (NVIDIA#2353)" This reverts commit 92c8482. * Add logs for missing CUDA and Cute. * autoformat --------- Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com> Signed-off-by: oliver könig <okoenig@nvidia.com> Signed-off-by: Ananth Subramaniam <ansubramania@nvidia.com> Signed-off-by: dimapihtar <dpihtar@gmail.com> Signed-off-by: Youngeun Kwon <youngeunk@nvidia.com> Signed-off-by: Youngeun <kyeg9404@gmail.com> Signed-off-by: Maanu Grover <maanug@nvidia.com> Signed-off-by: ykarnati <ykarnati@nvidia.com> Signed-off-by: Deepak Narayanan <dnarayanan@nvidia.com> Signed-off-by: GitHub Actions <github-actions[bot]@users.noreply.github.com> Signed-off-by: Charlie Truong <chtruong@nvidia.com> Signed-off-by: Zhongbo Zhu <zhongboz@nvidia.com> Signed-off-by: Xiaowei Ren <xren@nvidia.com> Signed-off-by: Xin Yao <xiny@nvidia.com> Signed-off-by: Keshav Santhanam <ksanthanam@nvidia.com> Signed-off-by: Pablo Garay <pagaray@nvidia.com> Signed-off-by: Asha Anoosheh <aanoosheh@nvidia.com> Signed-off-by: Chen Cui <chcui@nvidia.com> Signed-off-by: Li Tao <lit@nvidia.com> Signed-off-by: lit <lit@nvidia.com> Signed-off-by: Santosh Bhavani <santosh.bhavani@live.com> Signed-off-by: Robin Zhang <robinz@nvidia.com> Signed-off-by: kunlunl <kunlunl@nvidia.com> Co-authored-by: Deyu Fu <Deyu.Foo@gmail.com> Co-authored-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com> Co-authored-by: Yashaswi Karnati <144376261+yashaswikarnati@users.noreply.github.com> Co-authored-by: Jared Casper <155158+jaredcasper@users.noreply.github.com> Co-authored-by: Antoni-Joan Solergibert <asolergibert@nvidia.com> Co-authored-by: oliver könig <okoenig@nvidia.com> Co-authored-by: Ananth Subramaniam <ansubramania@nvidia.com> Co-authored-by: Teodor-Dumitru Ene <34819528+tdene@users.noreply.github.com> Co-authored-by: Siddharth Singh <136645615+sidsingh-nvidia@users.noreply.github.com> Co-authored-by: Mcore Bot <mcore-bot@nvidia.com> Co-authored-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com> Co-authored-by: Youngeun Kwon <youngeunk@nvidia.com> Co-authored-by: Lawrence McAfee <85179052+lmcafee-nvidia@users.noreply.github.com> Co-authored-by: Maanu Grover <109391026+maanug-nv@users.noreply.github.com> Co-authored-by: Lawrence McAfee <lmcafee@nvidia.com> Co-authored-by: AJ Schmidt <ajschmidt8@users.noreply.github.com> Co-authored-by: Deepak Narayanan <2724038+deepakn94@users.noreply.github.com> Co-authored-by: helen ngo <helenn@nvidia.com> Co-authored-by: GitHub Actions <github-actions[bot]@users.noreply.github.com> Co-authored-by: Aaron Gokaslan <aaronGokaslan@gmail.com> Co-authored-by: Robert Kirby <rkirby@nvidia.com> Co-authored-by: Teodor-Dumitru Ene <tene@nvidia.com> Co-authored-by: yeyu-nvidia <yeyu@nvidia.com> Co-authored-by: Abhinav Khattar <akhattar@nvidia.com> Co-authored-by: Roger Waleffe <rwaleffe@nvidia.com> Co-authored-by: Charlie Truong <chtruong@nvidia.com> Co-authored-by: Tong Liu <liutongt1998@gmail.com> Co-authored-by: Zhongbo Zhu <42691305+zhongbozhu@users.noreply.github.com> Co-authored-by: Xiaowei Ren <xren@nvidia.com> Co-authored-by: Xin Yao <xiny@nvidia.com> Co-authored-by: Teodor-Dumitru Ene <teodord.ene@gmail.com> Co-authored-by: Zijie Yan <zijiey@nvidia.com> Co-authored-by: root <root@pool0-01101.cm.cluster> Co-authored-by: Keshav Santhanam <ksanthanam@nvidia.com> Co-authored-by: Pablo Garay <pagaray@nvidia.com> Co-authored-by: Asha Anoosheh <aanoosheh@nvidia.com> Co-authored-by: Kan Zhu <kanz@nvidia.com> Co-authored-by: Robert Kirby <rkirby@cw-dfw-cs-001-vscode-01.cm.cluster> Co-authored-by: Jorge Albericio <jalbericiola@nvidia.com> Co-authored-by: Jon Barker <19699370+jon-barker@users.noreply.github.com> Co-authored-by: Chen Cui <chcui@nvidia.com> Co-authored-by: Pablo Garay <palenq@gmail.com> Co-authored-by: Tong Liu <tongliu@nvidia.com> Co-authored-by: Michael Wojcikiewicz <mwojcikiewic@nvidia.com> Co-authored-by: Li Tao <lit@nvidia.com> Co-authored-by: Santosh Bhavani <santosh.bhavani@live.com> Co-authored-by: Li Ruixiao <cgruixiao@outlook.com> Co-authored-by: Robin Zhang <robinz@nvidia.com> Co-authored-by: Kunlun Li <94586211+kunlunl@users.noreply.github.com>

fix: adding k8s taints for ephermeral jobs

fd27b23

theothermike requested a review from a team as a code owner November 27, 2025 18:53

copy-pr-bot bot temporarily deployed to nemo-ci November 27, 2025 18:54 Inactive

ko3n1g added this to the Core 0.16 milestone Nov 27, 2025

copy-pr-bot bot had a problem deploying to nemo-ci November 27, 2025 18:54 Failure

copy-pr-bot bot temporarily deployed to nemo-ci November 27, 2025 18:55 Inactive

copy-pr-bot bot temporarily deployed to test November 27, 2025 18:55 Inactive

ko3n1g approved these changes Nov 27, 2025

View reviewed changes

copy-pr-bot bot temporarily deployed to test November 27, 2025 20:14 Inactive

copy-pr-bot bot temporarily deployed to nemo-ci November 27, 2025 20:57 Inactive

theothermike added this pull request to the merge queue Nov 27, 2025

Merged via the queue into NVIDIA:main with commit b96d876 Nov 27, 2025
87 of 111 checks passed

theothermike deleted the theothermike/external-developers-isolation branch November 27, 2025 21:01

ko3n1g pushed a commit to ko3n1g/Megatron-LM that referenced this pull request Nov 28, 2025

fix: adding k8s taints for ephermeral jobs (NVIDIA#2420)

b750bdb

frsun-nvda pushed a commit to frsun-nvda/Megatron-LM that referenced this pull request Dec 1, 2025

fix: adding k8s taints for ephermeral jobs (NVIDIA#2420)

775aa1c

cspades pushed a commit to cspades/Megatron-LM that referenced this pull request Jan 7, 2026

fix: adding k8s taints for ephermeral jobs (NVIDIA#2420)

36ec511

daiyaanarfeen pushed a commit to daiyaanarfeen/Megatron-LM that referenced this pull request Feb 23, 2026

fix: adding k8s taints for ephermeral jobs (NVIDIA#2420)

558128d

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: adding k8s taints for ephermeral jobs#2420

fix: adding k8s taints for ephermeral jobs#2420
theothermike merged 1 commit intoNVIDIA:mainfrom
theothermike:theothermike/external-developers-isolation

theothermike commented Nov 27, 2025

Uh oh!

copy-pr-bot bot commented Nov 27, 2025

Uh oh!

theothermike commented Nov 27, 2025

Uh oh!

theothermike commented Nov 27, 2025

Uh oh!

theothermike commented Nov 27, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

theothermike commented Nov 27, 2025

What does this PR do ?

Contribution process

Pre-checks

Code review

(Step 1): Add PR label Expert Review

(Step 2): Collect the expert reviewers reviews

(Step 3): Final Review

(Optional Step 4): Cherry-pick into release branch

Merging your PR

Uh oh!

copy-pr-bot bot commented Nov 27, 2025

Uh oh!

theothermike commented Nov 27, 2025

Uh oh!

theothermike commented Nov 27, 2025

Uh oh!

theothermike commented Nov 27, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

(Step 1): Add PR label `Expert Review`