[None][fix] Fix GIL management for guided decoding host func by Tabrizian · Pull Request #13251 · NVIDIA/TensorRT-LLM

Tabrizian · 2026-04-21T01:08:34Z

Description

When the nano bind function passes Python objects by value we need to hold GIL for the entire duration of the function body since the Python objects will be destructed after the function completes.

Test Coverage

tests/integration/defs/accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_guided_decoding_with_eagle3 - exercises the guided decoding + eagle3 + disagg path that invokes launch_hostfunc.

PR Checklist

Please check this after reviewing the above items as appropriate for this PR.

Summary by CodeRabbit

Refactor
- Optimized runtime performance by improving host function execution efficiency

Tabrizian · 2026-04-21T01:08:47Z

/bot run --disable-fail-fast

Tabrizian · 2026-04-21T01:10:05Z

/bot run --disable-fail-fast

coderabbitai · 2026-04-21T01:12:41Z

📝 Walkthrough

Walkthrough

Removed explicit GIL acquisition and release guards from nanobind hostfunc bindings. The functions no longer acquire the GIL during construction and scheduling phases, while GIL acquisition remains in the trampoline during Python callback execution.

Changes

Cohort / File(s)	Summary
GIL Management `cpp/tensorrt_llm/nanobind/runtime/hostfunc.cpp`	Removed `nb::gil_scoped_acquire` from `launchHostFunc` start and `freeHostFuncUserData`. Removed `nb::call_guard<nb::gil_scoped_release>()` from `launch_hostfunc` and `free_hostfunc_user_data` binding declarations. GIL acquisition retained in callback trampoline execution.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly identifies the main change: fixing GIL management for the guided decoding host function, which aligns with the PR's core objective of simplifying GIL handling in hostfunc.cpp.
Description check	✅ Passed	PR description clearly explains the issue (GIL management in nanobind) and solution (removing manual GIL guards). Test coverage is specified with a concrete test path that exercises the modified code.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

tensorrt-cicd · 2026-04-21T01:14:42Z

PR_Github #44544 [ run ] triggered by Bot. Commit: e2dcbaf Link to invocation

tensorrt-cicd · 2026-04-21T01:17:47Z

PR_Github #44545 [ run ] triggered by Bot. Commit: e2dcbaf Link to invocation

tensorrt-cicd · 2026-04-21T09:03:02Z

PR_Github #44545 [ run ] completed with state SUCCESS. Commit: e2dcbaf
/LLM/main/L0_MergeRequest_PR pipeline #34939 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

Tabrizian · 2026-04-21T15:59:42Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-04-21T16:05:27Z

PR_Github #44756 [ run ] triggered by Bot. Commit: e2dcbaf Link to invocation

tensorrt-cicd · 2026-04-21T23:25:11Z

PR_Github #44756 [ run ] completed with state SUCCESS. Commit: e2dcbaf
/LLM/main/L0_MergeRequest_PR pipeline #35114 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

Tabrizian · 2026-04-22T00:31:58Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-04-22T00:38:01Z

PR_Github #44832 [ run ] triggered by Bot. Commit: 2e655d8 Link to invocation

mzweilz · 2026-04-22T09:47:37Z

/bot run

tensorrt-cicd · 2026-04-22T10:01:19Z

PR_Github #44950 [ run ] triggered by Bot. Commit: 2e655d8 Link to invocation

Tabrizian · 2026-04-29T15:34:55Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-04-29T15:41:47Z

PR_Github #46167 [ run ] triggered by Bot. Commit: 328e8c0 Link to invocation

tensorrt-cicd · 2026-04-30T04:06:38Z

PR_Github #46167 [ run ] completed with state SUCCESS. Commit: 328e8c0
/LLM/main/L0_MergeRequest_PR pipeline #36286 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Tabrizian · 2026-04-30T04:18:25Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-04-30T04:25:08Z

PR_Github #46291 [ run ] triggered by Bot. Commit: 328e8c0 Link to invocation

tensorrt-cicd · 2026-04-30T07:59:24Z

PR_Github #46291 [ run ] completed with state SUCCESS. Commit: 328e8c0
/LLM/main/L0_MergeRequest_PR pipeline #36392 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Tabrizian · 2026-04-30T16:19:16Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-04-30T16:25:35Z

PR_Github #46404 [ run ] triggered by Bot. Commit: 328e8c0 Link to invocation

tensorrt-cicd · 2026-05-01T03:10:56Z

PR_Github #46404 [ run ] completed with state SUCCESS. Commit: 328e8c0
/LLM/main/L0_MergeRequest_PR pipeline #36481 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Signed-off-by: Iman Tabrizian <[email protected]>

Tabrizian · 2026-05-12T18:56:55Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-05-12T19:02:35Z

PR_Github #48015 [ run ] triggered by Bot. Commit: cd1d9c5 Link to invocation

tensorrt-cicd · 2026-05-13T02:06:48Z

PR_Github #48015 [ run ] completed with state SUCCESS. Commit: cd1d9c5
/LLM/main/L0_MergeRequest_PR pipeline #37851 completed with status: 'SUCCESS'

CI Report

Link to invocation

…13251) Signed-off-by: Iman Tabrizian <[email protected]>

github-actions Bot assigned Tabrizian Apr 21, 2026

Tabrizian requested a review from syuoni April 21, 2026 01:09

Tabrizian enabled auto-merge (squash) April 21, 2026 01:09

Tabrizian disabled auto-merge April 21, 2026 01:10

Tabrizian changed the title ~~[None][fix] Simplify GIL management for guided decoding host func~~ [None][fix] Fix GIL management for guided decoding host func Apr 21, 2026

Tabrizian requested review from mikeiovine and sunnyqgg and removed request for syuoni April 21, 2026 16:35

mikeiovine requested a review from syuoni April 21, 2026 16:50

mikeiovine approved these changes Apr 21, 2026

View reviewed changes

Tabrizian force-pushed the user/imant/gil-hostfunc-main branch from e2dcbaf to 2e655d8 Compare April 22, 2026 00:31

Tabrizian force-pushed the user/imant/gil-hostfunc-main branch from 2e655d8 to 328e8c0 Compare April 29, 2026 15:34

Tabrizian enabled auto-merge (squash) April 29, 2026 15:35

Simplify GIL management for guided decoding host func

cd1d9c5

Signed-off-by: Iman Tabrizian <[email protected]>

Tabrizian force-pushed the user/imant/gil-hostfunc-main branch from 328e8c0 to cd1d9c5 Compare May 12, 2026 18:56

Tabrizian merged commit 550de9f into NVIDIA:main May 13, 2026
6 checks passed

yufeiwu-nv pushed a commit to yufeiwu-nv/TensorRT-LLM that referenced this pull request May 19, 2026

[None][fix] Fix GIL management for guided decoding host func (NVIDIA#…

49974c1

…13251) Signed-off-by: Iman Tabrizian <[email protected]>

Conversation

Tabrizian commented Apr 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Test Coverage

PR Checklist

Summary by CodeRabbit

Uh oh!

Tabrizian commented Apr 21, 2026

Uh oh!

Tabrizian commented Apr 21, 2026

Uh oh!

coderabbitai Bot commented Apr 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Estimated code review effort

❌ Failed checks (1 warning)

Uh oh!

tensorrt-cicd commented Apr 21, 2026

Uh oh!

tensorrt-cicd commented Apr 21, 2026

Uh oh!

tensorrt-cicd commented Apr 21, 2026

Uh oh!

Tabrizian commented Apr 21, 2026

Uh oh!

tensorrt-cicd commented Apr 21, 2026

Uh oh!

tensorrt-cicd commented Apr 21, 2026

Uh oh!

Tabrizian commented Apr 22, 2026

Uh oh!

tensorrt-cicd commented Apr 22, 2026

Uh oh!

mzweilz commented Apr 22, 2026

Uh oh!

tensorrt-cicd commented Apr 22, 2026

Uh oh!

Tabrizian commented Apr 29, 2026

Uh oh!

tensorrt-cicd commented Apr 29, 2026

Uh oh!

tensorrt-cicd commented Apr 30, 2026

Uh oh!

Tabrizian commented Apr 30, 2026

Uh oh!

tensorrt-cicd commented Apr 30, 2026

Uh oh!

tensorrt-cicd commented Apr 30, 2026

Uh oh!

Tabrizian commented Apr 30, 2026

Uh oh!

tensorrt-cicd commented Apr 30, 2026

Uh oh!

tensorrt-cicd commented May 1, 2026

Uh oh!

Tabrizian commented May 12, 2026

Uh oh!

tensorrt-cicd commented May 12, 2026

Uh oh!

tensorrt-cicd commented May 13, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Tabrizian commented Apr 21, 2026 •

edited

Loading

coderabbitai Bot commented Apr 21, 2026 •

edited

Loading