[None][fix] LlavaNext dtype fallback when text_config.torch_dtype is None by indrajit96 · Pull Request #12169 · NVIDIA/TensorRT-LLM

indrajit96 · 2026-03-12T20:52:37Z

Summary by CodeRabbit

Release Notes

Bug Fixes
- Enhanced model initialization with improved handling of data type configurations. The system now implements fallback mechanisms to ensure consistent behavior across input processors, vision models, and language model components. This prevents initialization errors and configuration-related issues, making the model loading process more reliable and stable across different deployment environments and hardware setups.

[https://nvbugspro.nvidia.com/bug/5790851][fix] Standalone Encoder EPD broken with Llava model

Description

A recent HuggingFace commit (2424fdd) removed torch_dtype: "bfloat16" from the text_config section of llava-hf/llava-v1.6-mistral-7b-hf's config.json.
This causes text_config.torch_dtype to resolve to None, which propagates through model initialization and ultimately triggers a KeyError: None in the KV cache manager when it tries to convert None to a dtype string via torch_dtype_to_str().

Fix: Fall back to the top-level config.torch_dtype whenever text_config.torch_dtype is None in three places:- LlavaNextInputProcessor.__init__ — self._dtype- LlavaNextVisionModel.__init__ — self.dtype- LlavaNextModel.__init__ — propagate torch_dtype to the LLM sub-config before constructing the language model

This is a defensive fix: HF model configs don't always replicate torch_dtype into every sub-config, and TRT-LLM should gracefully inherit from the parent config.

Test Coverage

PR Checklist

Please review the following before submitting your PR:

PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

Signed-off-by: Indrajit Bhosale <[email protected]>

indrajit96 · 2026-03-12T20:55:50Z

CC @chang-l @2ez4bz for review

coderabbitai · 2026-03-12T20:55:54Z

📝 Walkthrough

Walkthrough

The changes add fallback dtype initialization logic to three components in the LlaVA Next model: the input processor, vision model, and LLM model initialization. Each now uses a secondary dtype source (config.torch_dtype) when the primary source is unavailable.

Changes

Cohort / File(s)	Summary
LlaVA Next Model Dtype Initialization `tensorrt_llm/_torch/models/modeling_llava_next.py`	Added dtype fallback logic to input processor, vision model, and LLM model initialization. Each component now checks a secondary dtype source when the primary text_config.torch_dtype is unavailable.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~8 minutes

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Description check	✅ Passed	The PR description is well-structured with all required sections completed, including title format, detailed explanation of the issue and fix, and checklist verification.
Title check	✅ Passed	The title clearly and concisely summarizes the main change: adding dtype fallback logic for LlavaNext when text_config.torch_dtype is None, which matches the core fix described in the changeset.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

📝 Coding Plan

Generate coding plan for human review comments

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

Tip

You can disable sequence diagrams in the walkthrough.

Disable the reviews.sequence_diagrams setting to disable sequence diagrams in the walkthrough.

coderabbitai

Actionable comments posted: 1

🧹 Nitpick comments (1)

tensorrt_llm/_torch/models/modeling_llava_next.py (1)

63-64: Fallback logic is correct, but consider a guard for the double-None edge case.

The or fallback handles the primary scenario (HF omitting torch_dtype from text_config). However, if both config.text_config.torch_dtype and config.torch_dtype are None, self._dtype will remain None and could still trigger downstream errors.

Consider adding a defensive check or raising an informative error if the final dtype is None:

💡 Optional: Add explicit None guard

         self._dtype = (self.config.text_config.torch_dtype
                        or self.config.torch_dtype)
+        if self._dtype is None:
+            raise ValueError(
+                "Could not determine model dtype: both text_config.torch_dtype "
+                "and config.torch_dtype are None"
+            )

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@tensorrt_llm/_torch/models/modeling_llava_next.py` around lines 63 - 64, The
assignment to self._dtype uses a fallback (self.config.text_config.torch_dtype
or self.config.torch_dtype) but can still become None; update the initialization
in the class (where self._dtype is set) to guard this double-None case by either
raising a clear ValueError indicating both config.text_config.torch_dtype and
config.torch_dtype are unset, or by assigning a safe default (e.g.,
torch.float32); reference the symbols self._dtype,
config.text_config.torch_dtype, and config.torch_dtype and apply the check
immediately after the current fallback assignment.

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@tensorrt_llm/_torch/models/modeling_llava_next.py`:
- Around line 536-542: The code assigns llm_model_config.pretrained_config =
model_config.pretrained_config.text_config which creates a shared reference and
causes the later mutation (setting torch_dtype) to modify the original
model_config; to fix, assign a deep copy of the text sub-config instead (e.g.,
use copy.deepcopy on model_config.pretrained_config.text_config) before setting
torch_dtype so changes only affect llm_model_config.pretrained_config and do not
mutate model_config used elsewhere (refer to llm_model_config, model_config,
pretrained_config, text_config).

---

Nitpick comments:
In `@tensorrt_llm/_torch/models/modeling_llava_next.py`:
- Around line 63-64: The assignment to self._dtype uses a fallback
(self.config.text_config.torch_dtype or self.config.torch_dtype) but can still
become None; update the initialization in the class (where self._dtype is set)
to guard this double-None case by either raising a clear ValueError indicating
both config.text_config.torch_dtype and config.torch_dtype are unset, or by
assigning a safe default (e.g., torch.float32); reference the symbols
self._dtype, config.text_config.torch_dtype, and config.torch_dtype and apply
the check immediately after the current fallback assignment.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 6b870d6e-f75d-462d-8f1d-dd456f211c06

📥 Commits

Reviewing files that changed from the base of the PR and between 5cc0ccd and 268d64b.

📒 Files selected for processing (1)

tensorrt_llm/_torch/models/modeling_llava_next.py

indrajit96 · 2026-03-13T18:31:05Z

/bot run

indrajit96 · 2026-03-17T17:41:39Z

@chang-l @pengbowang-nv any update on this

chang-l · 2026-03-17T17:51:02Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-03-17T17:57:58Z

PR_Github #39302 [ run ] triggered by Bot. Commit: 268d64b Link to invocation

Signed-off-by: Indrajit Bhosale <[email protected]>

tensorrt-cicd · 2026-03-17T19:25:27Z

PR_Github #39302 [ run ] completed with state SUCCESS. Commit: 268d64b
/LLM/main/L0_MergeRequest_PR pipeline #30552 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

indrajit96 · 2026-03-18T16:55:32Z

/bot run --disable-fail-fast

indrajit96 · 2026-03-18T17:05:32Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-03-18T17:16:43Z

PR_Github #39489 [ run ] triggered by Bot. Commit: 70ba420 Link to invocation

tensorrt-cicd · 2026-03-18T21:35:22Z

PR_Github #39489 [ run ] completed with state SUCCESS. Commit: 70ba420
/LLM/main/L0_MergeRequest_PR pipeline #30715 completed with status: 'SUCCESS'

CI Report

Link to invocation

indrajit96 · 2026-03-18T21:46:06Z

@chang-l @Wanli-Jiang Can we merge this CI is green?

…None (NVIDIA#12169) Signed-off-by: Indrajit Bhosale <[email protected]>

Fix LlavaNext dtype fallback when text_config.torch_dtype is None

268d64b

Signed-off-by: Indrajit Bhosale <[email protected]>

indrajit96 requested a review from a team as a code owner March 12, 2026 20:52

indrajit96 requested a review from Wanli-Jiang March 12, 2026 20:52

coderabbitai Bot reviewed Mar 12, 2026

View reviewed changes

Comment thread tensorrt_llm/_torch/models/modeling_llava_next.py

nv-yna requested a review from chang-l March 12, 2026 21:09

svc-trtllm-gh-bot added the Community want to contribute PRs initiated from Community label Mar 12, 2026

pengbowang-nv removed the Community want to contribute PRs initiated from Community label Mar 13, 2026

2ez4bz approved these changes Mar 13, 2026

View reviewed changes

svc-trtllm-gh-bot added the Community want to contribute PRs initiated from Community label Mar 13, 2026

chang-l changed the title ~~fix: LlavaNext dtype fallback when text_config.torch_dtype is None~~ [None][fix] LlavaNext dtype fallback when text_config.torch_dtype is None Mar 17, 2026

[None][fix] Fix pre-commit formatting for LlavaNext dtype fallback

ed5510f

Signed-off-by: Indrajit Bhosale <[email protected]>

Merge branch 'main' into fix/llava-next-dtype-fallback

7b240ac

Merge branch 'main' into fix/llava-next-dtype-fallback

70ba420

chang-l merged commit 58ec688 into NVIDIA:main Mar 18, 2026
5 checks passed

chang-l removed the Community want to contribute PRs initiated from Community label Mar 18, 2026

limin2021 pushed a commit to limin2021/TensorRT-LLM that referenced this pull request Mar 19, 2026

[None][fix] LlavaNext dtype fallback when text_config.torch_dtype is …

f92bab8

…None (NVIDIA#12169) Signed-off-by: Indrajit Bhosale <[email protected]>

indrajit96 mentioned this pull request Mar 24, 2026

test: Add ci coverage for trtllm multimodal raw embeddings ai-dynamo/dynamo#7540

Merged

longcheng-nv pushed a commit to longcheng-nv/TensorRT-LLM that referenced this pull request Mar 31, 2026

[None][fix] LlavaNext dtype fallback when text_config.torch_dtype is …

d1093c2

…None (NVIDIA#12169) Signed-off-by: Indrajit Bhosale <[email protected]>

Conversation

indrajit96 commented Mar 12, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by CodeRabbit

Release Notes

Description

Test Coverage

PR Checklist

GitHub Bot Help

Uh oh!

indrajit96 commented Mar 12, 2026

Uh oh!

coderabbitai Bot commented Mar 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Estimated code review effort

❌ Failed checks (1 warning)

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

indrajit96 commented Mar 13, 2026

Uh oh!

indrajit96 commented Mar 17, 2026

Uh oh!

chang-l commented Mar 17, 2026

Uh oh!

tensorrt-cicd commented Mar 17, 2026

Uh oh!

tensorrt-cicd commented Mar 17, 2026

Uh oh!

indrajit96 commented Mar 18, 2026

Uh oh!

indrajit96 commented Mar 18, 2026

Uh oh!

tensorrt-cicd commented Mar 18, 2026

Uh oh!

tensorrt-cicd commented Mar 18, 2026

Uh oh!

indrajit96 commented Mar 18, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

indrajit96 commented Mar 12, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Mar 12, 2026 •

edited

Loading