[None][test] add Nemotron Ultra V3 AutoDeploy accuracy test#13658
Conversation
bf4fba1 to
5f905e2
Compare
|
/bot run --extra-stage "DGX_B200-4_GPUs-AutoDeploy-1, DGX_H100-4_GPUs-AutoDeploy-1" |
📝 WalkthroughWalkthroughThis pull request adds support for the Nemotron-Ultra-V3 model by introducing model registry configuration, accuracy reference benchmarks for GSM8K and MMLU, an integration test class with parametrized testing across multiple backends and configurations, test list entries, and HuggingFace Hub ID mapping support. Changes
Estimated code review effort🎯 2 (Simple) | ⏱️ ~12 minutes 🚥 Pre-merge checks | ✅ 3 | ❌ 2❌ Failed checks (2 warnings)
✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Warning Review ran into problems🔥 ProblemsTimed out fetching pipeline failures after 30000ms Review rate limit: 8/10 reviews remaining, refill in 10 minutes and 39 seconds. Comment |
|
PR_Github #46356 [ run ] triggered by Bot. Commit: |
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
tests/integration/test_lists/test-db/l0_dgx_b200.yml (1)
326-335:⚠️ Potential issue | 🟠 Major | ⚡ Quick winAdd a pre-merge
trtllmUltra-V3 selection.The new model-registry path is trtllm-oriented, but this pre-merge block only schedules the flashinfer variant. That leaves the primary runtime unvalidated until post-merge.
Suggested fix
- accuracy/test_llm_api_autodeploy.py::TestNemotronUltraV3::test_accuracy[nvfp4-4-attn_dp_off-flashinfer] + - accuracy/test_llm_api_autodeploy.py::TestNemotronUltraV3::test_accuracy[nvfp4-4-attn_dp_off-trtllm]🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@tests/integration/test_lists/test-db/l0_dgx_b200.yml` around lines 326 - 335, The pre-merge test list schedules only the flashinfer variant for Ultra-V3, leaving the new trtllm-oriented model-registry path untested; update the pre-merge block in this YAML to include a trtllm Ultra-V3 entry (mirror the existing accuracy/test_llm_api_autodeploy.py::TestNemotronUltraV3::test_accuracy[nvfp4-4-attn_dp_off-flashinfer] line) so the corresponding trtllm runtime is validated pre-merge, and ensure any grouping keys or tags used for pre-merge selection reference the trtllm variant as well.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@tests/integration/defs/accuracy/test_llm_api_autodeploy.py`:
- Around line 722-785: The test exercises only the NVFP4 path
(MODEL_PATHS["nvfp4"]) so add an early guard in
TestNemotronUltraV3.test_accuracy to skip when the host GPUs are not
Blackwell/B200-class; specifically, after the existing get_device_count() check,
call the project helper that detects Blackwell/B200 GPUs (e.g., is_blackwell()
or the equivalent GPU-family detection helper) and pytest.skip with a
descriptive message if it returns False, ensuring the test only runs on
Blackwell systems.
---
Outside diff comments:
In `@tests/integration/test_lists/test-db/l0_dgx_b200.yml`:
- Around line 326-335: The pre-merge test list schedules only the flashinfer
variant for Ultra-V3, leaving the new trtllm-oriented model-registry path
untested; update the pre-merge block in this YAML to include a trtllm Ultra-V3
entry (mirror the existing
accuracy/test_llm_api_autodeploy.py::TestNemotronUltraV3::test_accuracy[nvfp4-4-attn_dp_off-flashinfer]
line) so the corresponding trtllm runtime is validated pre-merge, and ensure any
grouping keys or tags used for pre-merge selection reference the trtllm variant
as well.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 46ca0eed-98ad-45c6-95c9-766a12295a41
📒 Files selected for processing (6)
examples/auto_deploy/model_registry/configs/ultra_v3.yamltests/integration/defs/accuracy/references/gsm8k.yamltests/integration/defs/accuracy/references/mmlu.yamltests/integration/defs/accuracy/test_llm_api_autodeploy.pytests/integration/test_lists/test-db/l0_dgx_b200.ymltests/test_common/llm_data.py
|
PR_Github #46356 [ run ] completed with state
|
5f905e2 to
eee2b19
Compare
|
/bot run --extra-stage "DGX_B200-4_GPUs-AutoDeploy-1, DGX_H100-4_GPUs-AutoDeploy-1" |
|
PR_Github #46585 [ run ] triggered by Bot. Commit: |
|
PR_Github #46585 [ run ] completed with state
|
|
/bot run --extra-stage "DGX_B200-4_GPUs-AutoDeploy-1, DGX_H100-4_GPUs-AutoDeploy-1" |
|
PR_Github #46927 [ run ] triggered by Bot. Commit: |
|
PR_Github #46927 [ run ] completed with state
|
|
/bot run --extra-stage "DGX_B200-4_GPUs-AutoDeploy-1, DGX_H100-4_GPUs-AutoDeploy-1" --reuse-test |
|
PR_Github #46994 [ run ] triggered by Bot. Commit: |
|
PR_Github #46994 [ run ] completed with state
|
|
/bot run --extra-stage "DGX_B200-4_GPUs-AutoDeploy-1, DGX_H100-4_GPUs-AutoDeploy-1" --reuse-test |
eee2b19 to
7f0b29f
Compare
|
PR_Github #47126 [ run ] triggered by Bot. Commit: |
|
@coderabbitai review |
✅ Actions performedReview triggered.
|
|
PR_Github #47126 [ run ] completed with state
|
|
PR_Github #47727 [ run ] triggered by Bot. Commit: |
|
PR_Github #47727 [ run ] completed with state
|
|
/bot run --extra-stage "DGX_B200-4_GPUs-AutoDeploy-1" --reuse-test |
|
PR_Github #47749 [ run ] triggered by Bot. Commit: |
|
PR_Github #47749 [ run ] completed with state
|
Signed-off-by: Tal Cherckez <[email protected]>
Signed-off-by: Tal Cherckez <[email protected]>
Signed-off-by: Tal Cherckez <[email protected]>
a7d298a to
312bdb8
Compare
|
/bot run --extra-stage "DGX_B200-4_GPUs-AutoDeploy-1" --reuse-test |
|
PR_Github #47894 [ run ] triggered by Bot. Commit: |
|
PR_Github #47894 [ run ] completed with state
|
|
/bot run --extra-stage "DGX_B200-4_GPUs-AutoDeploy-1" --reuse-test |
|
PR_Github #47959 [ run ] triggered by Bot. Commit: |
|
PR_Github #47959 [ run ] completed with state
|
|
/bot run help |
|
/bot help |
GitHub Bot Help
Provide a user friendly way for developers to interact with a Jenkins server. Run See details below for each supported subcommand. Details
Launch build/test pipelines. All previously running jobs will be killed.
kill
Kill all running builds associated with pull request. skip
Skip testing for latest commit on pull request. reuse-pipeline
Reuse a previous pipeline to validate current commit. This action will also kill all currently running builds associated with the pull request. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break. |
|
/bot run --stage-list "DGX_B200-4_GPUs-AutoDeploy-1" |
|
PR_Github #47986 Bot args parsing error: usage: /bot [-h] |
|
PR_Github #47987 [ run ] triggered by Bot. Commit: |
|
PR_Github #47987 [ run ] completed with state |
|
/bot run |
|
PR_Github #48000 [ run ] triggered by Bot. Commit: |
|
PR_Github #48000 [ run ] completed with state |
…3658) Signed-off-by: Tal Cherckez <[email protected]>
Summary by CodeRabbit
New Features
Tests
Description
Test Coverage
PR Checklist
Please review the following before submitting your PR:
PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.
GitHub Bot Help
To see a list of available CI bot commands, please comment
/bot help.