Skip to content

[None][test] add Nemotron Ultra V3 AutoDeploy accuracy test#13658

Merged
tcherckez-nvidia merged 3 commits into
NVIDIA:mainfrom
nv-auto-deploy:ultra-v3-tests
May 12, 2026
Merged

[None][test] add Nemotron Ultra V3 AutoDeploy accuracy test#13658
tcherckez-nvidia merged 3 commits into
NVIDIA:mainfrom
nv-auto-deploy:ultra-v3-tests

Conversation

@tcherckez-nvidia
Copy link
Copy Markdown
Collaborator

@tcherckez-nvidia tcherckez-nvidia commented Apr 30, 2026

Summary by CodeRabbit

  • New Features

    • Introduced Nemotron-Ultra-V3 model with optimized performance configurations for enhanced inference capabilities.
    • Established accuracy benchmarks across standard evaluation datasets.
  • Tests

    • Added comprehensive integration accuracy tests spanning multiple hardware configurations and inference backends.
    • Implemented test coverage for both single and distributed multi-GPU deployment scenarios.

Description

Test Coverage

PR Checklist

Please review the following before submitting your PR:

  • PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

  • PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

  • Test cases are provided for new code paths (see test instructions)

  • Any new dependencies have been scanned for license and vulnerabilities

  • CODEOWNERS updated if ownership changes

  • Documentation updated as needed

  • Update tava architecture diagram if there is a significant design change in PR.

  • The reviewers assigned automatically/manually are appropriate for the PR.

  • Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

@tcherckez-nvidia
Copy link
Copy Markdown
Collaborator Author

/bot run --extra-stage "DGX_B200-4_GPUs-AutoDeploy-1, DGX_H100-4_GPUs-AutoDeploy-1"

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented Apr 30, 2026

📝 Walkthrough

Walkthrough

This pull request adds support for the Nemotron-Ultra-V3 model by introducing model registry configuration, accuracy reference benchmarks for GSM8K and MMLU, an integration test class with parametrized testing across multiple backends and configurations, test list entries, and HuggingFace Hub ID mapping support.

Changes

Cohort / File(s) Summary
Model Configuration
examples/auto_deploy/model_registry/configs/ultra_v3.yaml
New model registry config for ultra_v3 with TensorRT-LLM runtime, CUDA graph compilation, chunked prefill, mamba attention backend, and compilation optimizations including MoE fusion, SSM attention caching, and sharding detection via SYMM_MEM allreduce strategy.
Accuracy References
tests/integration/defs/accuracy/references/gsm8k.yaml, tests/integration/defs/accuracy/references/mmlu.yaml
New accuracy reference entries for nvidia/Nemotron-Ultra-V3 with NVFP4 quantization and FP8 KV-cache quantization, specifying GSM8K accuracy of 91.797 and MMLU accuracy of 85.70.
Integration Tests
tests/integration/defs/accuracy/test_llm_api_autodeploy.py, tests/integration/test_lists/test-db/l0_dgx_b200.yml
New test class TestNemotronUltraV3 with parametrized accuracy tests across flashinfer and trtllm backends, attention-DP sharding modes, and multi-GPU world sizes (4, 8); corresponding test list entries added for l0_dgx_b200 configurations.
Data Mapping
tests/test_common/llm_data.py
Added HuggingFace Hub ID mapping for nvidia/Nemotron-Ultra-V3-NVFP4 to local model directory resolution in mock_snapshot_download function.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~12 minutes

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name Status Explanation Resolution
Description check ⚠️ Warning The PR description is incomplete; it contains only the template structure with empty sections for Description and Test Coverage, and lacks actual implementation details. Fill in the Description section explaining what test is being added and why, and the Test Coverage section listing the relevant test cases added (e.g., TestNemotronUltraV3 with MMLU/GSM8K accuracy tests).
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (3 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly summarizes the main change: adding an AutoDeploy accuracy test for Nemotron Ultra V3 model.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Warning

Review ran into problems

🔥 Problems

Timed out fetching pipeline failures after 30000ms


Review rate limit: 8/10 reviews remaining, refill in 10 minutes and 39 seconds.

Comment @coderabbitai help to get the list of available commands and usage tips.

@tensorrt-cicd
Copy link
Copy Markdown
Collaborator

PR_Github #46356 [ run ] triggered by Bot. Commit: 5f905e2 Link to invocation

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
tests/integration/test_lists/test-db/l0_dgx_b200.yml (1)

326-335: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Add a pre-merge trtllm Ultra-V3 selection.

The new model-registry path is trtllm-oriented, but this pre-merge block only schedules the flashinfer variant. That leaves the primary runtime unvalidated until post-merge.

Suggested fix
   - accuracy/test_llm_api_autodeploy.py::TestNemotronUltraV3::test_accuracy[nvfp4-4-attn_dp_off-flashinfer]
+  - accuracy/test_llm_api_autodeploy.py::TestNemotronUltraV3::test_accuracy[nvfp4-4-attn_dp_off-trtllm]
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@tests/integration/test_lists/test-db/l0_dgx_b200.yml` around lines 326 - 335,
The pre-merge test list schedules only the flashinfer variant for Ultra-V3,
leaving the new trtllm-oriented model-registry path untested; update the
pre-merge block in this YAML to include a trtllm Ultra-V3 entry (mirror the
existing
accuracy/test_llm_api_autodeploy.py::TestNemotronUltraV3::test_accuracy[nvfp4-4-attn_dp_off-flashinfer]
line) so the corresponding trtllm runtime is validated pre-merge, and ensure any
grouping keys or tags used for pre-merge selection reference the trtllm variant
as well.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@tests/integration/defs/accuracy/test_llm_api_autodeploy.py`:
- Around line 722-785: The test exercises only the NVFP4 path
(MODEL_PATHS["nvfp4"]) so add an early guard in
TestNemotronUltraV3.test_accuracy to skip when the host GPUs are not
Blackwell/B200-class; specifically, after the existing get_device_count() check,
call the project helper that detects Blackwell/B200 GPUs (e.g., is_blackwell()
or the equivalent GPU-family detection helper) and pytest.skip with a
descriptive message if it returns False, ensuring the test only runs on
Blackwell systems.

---

Outside diff comments:
In `@tests/integration/test_lists/test-db/l0_dgx_b200.yml`:
- Around line 326-335: The pre-merge test list schedules only the flashinfer
variant for Ultra-V3, leaving the new trtllm-oriented model-registry path
untested; update the pre-merge block in this YAML to include a trtllm Ultra-V3
entry (mirror the existing
accuracy/test_llm_api_autodeploy.py::TestNemotronUltraV3::test_accuracy[nvfp4-4-attn_dp_off-flashinfer]
line) so the corresponding trtllm runtime is validated pre-merge, and ensure any
grouping keys or tags used for pre-merge selection reference the trtllm variant
as well.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 46ca0eed-98ad-45c6-95c9-766a12295a41

📥 Commits

Reviewing files that changed from the base of the PR and between 24228e9 and 5f905e2.

📒 Files selected for processing (6)
  • examples/auto_deploy/model_registry/configs/ultra_v3.yaml
  • tests/integration/defs/accuracy/references/gsm8k.yaml
  • tests/integration/defs/accuracy/references/mmlu.yaml
  • tests/integration/defs/accuracy/test_llm_api_autodeploy.py
  • tests/integration/test_lists/test-db/l0_dgx_b200.yml
  • tests/test_common/llm_data.py

Comment thread tests/integration/defs/accuracy/test_llm_api_autodeploy.py
@tensorrt-cicd
Copy link
Copy Markdown
Collaborator

PR_Github #46356 [ run ] completed with state SUCCESS. Commit: 5f905e2
/LLM/main/L0_MergeRequest_PR pipeline #36443 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Comment thread tests/integration/defs/accuracy/test_llm_api_autodeploy.py Outdated
Comment thread tests/integration/defs/accuracy/test_llm_api_autodeploy.py
Comment thread tests/integration/test_lists/test-db/l0_dgx_b200.yml Outdated
Comment thread tests/integration/test_lists/test-db/l0_dgx_b200.yml Outdated
@tcherckez-nvidia
Copy link
Copy Markdown
Collaborator Author

/bot run --extra-stage "DGX_B200-4_GPUs-AutoDeploy-1, DGX_H100-4_GPUs-AutoDeploy-1"

@tensorrt-cicd
Copy link
Copy Markdown
Collaborator

PR_Github #46585 [ run ] triggered by Bot. Commit: eee2b19 Link to invocation

@tensorrt-cicd
Copy link
Copy Markdown
Collaborator

PR_Github #46585 [ run ] completed with state SUCCESS. Commit: eee2b19
/LLM/main/L0_MergeRequest_PR pipeline #36634 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@tcherckez-nvidia
Copy link
Copy Markdown
Collaborator Author

/bot run --extra-stage "DGX_B200-4_GPUs-AutoDeploy-1, DGX_H100-4_GPUs-AutoDeploy-1"

@tensorrt-cicd
Copy link
Copy Markdown
Collaborator

PR_Github #46927 [ run ] triggered by Bot. Commit: eee2b19 Link to invocation

@tensorrt-cicd
Copy link
Copy Markdown
Collaborator

PR_Github #46927 [ run ] completed with state SUCCESS. Commit: eee2b19
/LLM/main/L0_MergeRequest_PR pipeline #36933 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

@tcherckez-nvidia
Copy link
Copy Markdown
Collaborator Author

/bot run --extra-stage "DGX_B200-4_GPUs-AutoDeploy-1, DGX_H100-4_GPUs-AutoDeploy-1" --reuse-test

@tensorrt-cicd
Copy link
Copy Markdown
Collaborator

PR_Github #46994 [ run ] triggered by Bot. Commit: eee2b19 Link to invocation

@tcherckez-nvidia tcherckez-nvidia enabled auto-merge (squash) May 6, 2026 14:13
@tensorrt-cicd
Copy link
Copy Markdown
Collaborator

PR_Github #46994 [ run ] completed with state SUCCESS. Commit: eee2b19
/LLM/main/L0_MergeRequest_PR pipeline #36969 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@tcherckez-nvidia
Copy link
Copy Markdown
Collaborator Author

/bot run --extra-stage "DGX_B200-4_GPUs-AutoDeploy-1, DGX_H100-4_GPUs-AutoDeploy-1" --reuse-test

@tensorrt-cicd
Copy link
Copy Markdown
Collaborator

PR_Github #47126 [ run ] triggered by Bot. Commit: 7f0b29f Link to invocation

@xinhe-nv
Copy link
Copy Markdown
Collaborator

xinhe-nv commented May 7, 2026

@coderabbitai review

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented May 7, 2026

✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

Comment thread tests/integration/test_lists/test-db/l0_dgx_b200.yml
@tensorrt-cicd
Copy link
Copy Markdown
Collaborator

PR_Github #47126 [ run ] completed with state SUCCESS. Commit: 7f0b29f
/LLM/main/L0_MergeRequest_PR pipeline #37091 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@tensorrt-cicd
Copy link
Copy Markdown
Collaborator

PR_Github #47727 [ run ] triggered by Bot. Commit: a7d298a Link to invocation

@tensorrt-cicd
Copy link
Copy Markdown
Collaborator

PR_Github #47727 [ run ] completed with state SUCCESS. Commit: a7d298a
/LLM/main/L0_MergeRequest_PR pipeline #37623 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@tcherckez-nvidia
Copy link
Copy Markdown
Collaborator Author

/bot run --extra-stage "DGX_B200-4_GPUs-AutoDeploy-1" --reuse-test

@tensorrt-cicd
Copy link
Copy Markdown
Collaborator

PR_Github #47749 [ run ] triggered by Bot. Commit: a7d298a Link to invocation

@tensorrt-cicd
Copy link
Copy Markdown
Collaborator

PR_Github #47749 [ run ] completed with state SUCCESS. Commit: a7d298a
/LLM/main/L0_MergeRequest_PR pipeline #37643 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@tcherckez-nvidia
Copy link
Copy Markdown
Collaborator Author

/bot run --extra-stage "DGX_B200-4_GPUs-AutoDeploy-1" --reuse-test

@tensorrt-cicd
Copy link
Copy Markdown
Collaborator

PR_Github #47894 [ run ] triggered by Bot. Commit: 312bdb8 Link to invocation

@tensorrt-cicd
Copy link
Copy Markdown
Collaborator

PR_Github #47894 [ run ] completed with state SUCCESS. Commit: 312bdb8
/LLM/main/L0_MergeRequest_PR pipeline #37745 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@tcherckez-nvidia
Copy link
Copy Markdown
Collaborator Author

/bot run --extra-stage "DGX_B200-4_GPUs-AutoDeploy-1" --reuse-test

@tensorrt-cicd
Copy link
Copy Markdown
Collaborator

PR_Github #47959 [ run ] triggered by Bot. Commit: 312bdb8 Link to invocation

@tensorrt-cicd
Copy link
Copy Markdown
Collaborator

PR_Github #47959 [ run ] completed with state SUCCESS. Commit: 312bdb8
/LLM/main/L0_MergeRequest_PR pipeline #37800 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@tcherckez-nvidia
Copy link
Copy Markdown
Collaborator Author

/bot run help

@tcherckez-nvidia
Copy link
Copy Markdown
Collaborator Author

/bot help

@github-actions
Copy link
Copy Markdown

GitHub Bot Help

/bot [-h] ['run', 'kill', 'skip', 'reuse-pipeline'] ...

Provide a user friendly way for developers to interact with a Jenkins server.

Run /bot [-h|--help] to print this help message.

See details below for each supported subcommand.

Details

run [--reuse-test (optional)pipeline-id --disable-fail-fast --skip-test --stage-list "A10-PyTorch-1, xxx" --gpu-type "A30, H100_PCIe" --test-backend "pytorch, cpp" --add-multi-gpu-test --only-multi-gpu-test --disable-multi-gpu-test --post-merge --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx" --detailed-log --debug(experimental) --high-priority]

Launch build/test pipelines. All previously running jobs will be killed.

--reuse-test (optional)pipeline-id (OPTIONAL) : Allow the new pipeline to reuse build artifacts and skip successful test stages from a specified pipeline or the last pipeline if no pipeline-id is indicated. If the Git commit ID has changed, this option will be always ignored. The DEFAULT behavior of the bot is to reuse build artifacts and successful test results from the last pipeline.

--disable-reuse-test (OPTIONAL) : Explicitly prevent the pipeline from reusing build artifacts and skipping successful test stages from a previous pipeline. Ensure that all builds and tests are run regardless of previous successes.

--disable-fail-fast (OPTIONAL) : Disable fail fast on build/tests/infra failures.

--skip-test (OPTIONAL) : Skip all test stages, but still run build stages, package stages and sanity check stages. Note: Does NOT update GitHub check status.

--stage-list "A10-PyTorch-1, xxx" (OPTIONAL) : Only run the specified test stages. Supports wildcard * for pattern matching (e.g., "*PerfSanity*" matches all stages containing PerfSanity). Examples: "A10-PyTorch-1, xxx", "PerfSanity". Note: Does NOT update GitHub check status.

--gpu-type "A30, H100_PCIe" (OPTIONAL) : Only run the test stages on the specified GPU types. Examples: "A30, H100_PCIe". Note: Does NOT update GitHub check status.

--test-backend "pytorch, cpp" (OPTIONAL) : Skip test stages which don't match the specified backends. Only support [pytorch, cpp, tensorrt, triton]. Examples: "pytorch, cpp" (does not run test stages with tensorrt or triton backend). Note: Does NOT update GitHub pipeline status.

--only-multi-gpu-test (OPTIONAL) : Only run the multi-GPU tests. Note: Does NOT update GitHub check status.

--disable-multi-gpu-test (OPTIONAL) : Disable the multi-GPU tests. Note: Does NOT update GitHub check status.

--add-multi-gpu-test (OPTIONAL) : Force run the multi-GPU tests in addition to running L0 pre-merge pipeline.

--post-merge (OPTIONAL) : Run the L0 post-merge pipeline instead of the ordinary L0 pre-merge pipeline.

--extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx" (OPTIONAL) : Run the ordinary L0 pre-merge pipeline and specified test stages. Supports wildcard * for pattern matching. Examples: --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx", --extra-stage "Post-Merge".

--detailed-log (OPTIONAL) : Enable flushing out all logs to the Jenkins console. This will significantly increase the log volume and may slow down the job.

--debug (OPTIONAL) : Experimental feature. Enable access to the CI container for debugging purpose. Note: Specify exactly one stage in the stage-list parameter to access the appropriate container environment. Note: Does NOT update GitHub check status.

--high-priority (OPTIONAL) : Run the pipeline with high priority. This option is restricted to authorized users only and will route the job to a high-priority queue.

kill

kill

Kill all running builds associated with pull request.

skip

skip --comment COMMENT

Skip testing for latest commit on pull request. --comment "Reason for skipping build/test" is required. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

reuse-pipeline

reuse-pipeline

Reuse a previous pipeline to validate current commit. This action will also kill all currently running builds associated with the pull request. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

@tcherckez-nvidia
Copy link
Copy Markdown
Collaborator Author

/bot run --stage-list "DGX_B200-4_GPUs-AutoDeploy-1"

@tensorrt-cicd
Copy link
Copy Markdown
Collaborator

PR_Github #47986 Bot args parsing error: usage: /bot [-h]
{run,kill,skip,submit,reviewers,reuse-pipeline,reuse-review} ...
/bot: error: unrecognized arguments: help

Link to invocation

@tensorrt-cicd
Copy link
Copy Markdown
Collaborator

PR_Github #47987 [ run ] triggered by Bot. Commit: 312bdb8 Link to invocation

@tensorrt-cicd
Copy link
Copy Markdown
Collaborator

PR_Github #47987 [ run ] completed with state SUCCESS. Commit: 312bdb8
/LLM/main/L0_MergeRequest_PR pipeline #37826 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

@tcherckez-nvidia
Copy link
Copy Markdown
Collaborator Author

/bot run

@tensorrt-cicd
Copy link
Copy Markdown
Collaborator

PR_Github #48000 [ run ] triggered by Bot. Commit: 312bdb8 Link to invocation

@tensorrt-cicd
Copy link
Copy Markdown
Collaborator

PR_Github #48000 [ run ] completed with state SUCCESS. Commit: 312bdb8
/LLM/main/L0_MergeRequest_PR pipeline #37837 completed with status: 'SUCCESS'

CI Report

Link to invocation

@tcherckez-nvidia tcherckez-nvidia merged commit 113bc64 into NVIDIA:main May 12, 2026
6 checks passed
yufeiwu-nv pushed a commit to yufeiwu-nv/TensorRT-LLM that referenced this pull request May 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants