Fix warm shutdown RuntimeError with eventlet>=0.37.0 (#10083) by ChickenBenny · Pull Request #10123 · celery/celery

ChickenBenny · 2026-02-15T04:40:30Z

Description

Fixes #10083 - Celery warm shutdown fails with RuntimeError: do not call blocking functions from the mainloop when using celery>=5.5.0b3 and eventlet>=0.37.0.

Problem

When pressing Ctrl+C to initiate a warm shutdown with the eventlet pool, users encounter:

RuntimeError: do not call blocking functions from the mainloop

Root Cause

PR print is not thread-safe, so should not be used in signal handler #9222 changed safe_say() to use os.write() instead of print() for signal-safe output
Eventlet 0.37.0 (PR #975) changed os.write() to use cooperative I/O via hubs.trampoline()
When SIGINT is received, the signal handler runs inside eventlet's hub event loop
Calling the patched os.write() from within the hub triggers the RuntimeError

Call Stack

SIGINT received →
  hub.run() [eventlet mainloop] →
    _handle_request() →
      on_SIGINT() →
        safe_say() →
          os.write() [eventlet-patched] →
            hubs.trampoline() →
              RuntimeError: do not call blocking functions from the mainloop

Solution

Save the original os.write function in celery/__init__.py before maybe_patch_concurrency() is called (which triggers eventlet's monkey-patching), and import it in celery/apps/worker.py for use in safe_say():

# celery/__init__.py - after all imports, before maybe_patch_concurrency
_original_os_write = os.write

# celery/apps/worker.py
from celery import _original_os_write

def safe_say(msg, f=sys.__stderr__):
    if hasattr(f, 'fileno') and f.fileno() is not None:
        _original_os_write(f.fileno(), f'\n{msg}\n'.encode())

Why This Works

celery/__init__.py is imported before maybe_patch_concurrency() is called
maybe_patch_concurrency() calls eventlet.monkey_patch() which patches os.write
celery/apps/worker.py is imported after monkey-patching, but it uses the pre-saved original
The original os.write() is async-signal-safe and non-blocking
It doesn't interact with eventlet's hub/greenlet scheduler
Signal handlers can safely call it without triggering hub conflicts

Changes

celery/__init__.py:
- Save _original_os_write = os.write at module level after all imports
- Export via local.recreate_module() for lazy loading compatibility
- Add comment explaining the purpose
celery/apps/worker.py:
- Import _original_os_write from celery
- Update safe_say() to use _original_os_write instead of os.write
- Add docstring explaining the rationale

Testing

Manual Testing

Install eventlet>=0.37.0
Start worker: celery -A myapp worker --pool=eventlet
Press Ctrl+C
Before fix: RuntimeError
After fix: Clean warm shutdown message

Reproduction Script

Create a test file to reproduce the issue:

# test_eventlet_shutdown.py
"""Simple Celery app for testing eventlet warm shutdown fix."""
from celery import Celery

# Create a simple Celery app
app = Celery('test_eventlet_shutdown')

# Configure broker (using memory transport for simplicity)
app.conf.update(
    broker_url='memory://',
    result_backend='cache+memory://',
    task_always_eager=False,  # Don't execute tasks immediately
    worker_prefetch_multiplier=1,
    task_acks_late=True,
)

@app.task
def simple_task(x, y):
    """A simple task that adds two numbers."""
    import time
    time.sleep(2)  # Simulate some work
    return x + y

@app.task
def long_running_task():
    """A longer task to test shutdown during execution."""
    import time
    print("Starting long running task...")
    for i in range(10):
        print(f"Working... {i+1}/10")
        time.sleep(1)
    print("Long running task completed!")
    return "done"

if __name__ == '__main__':
    app.start()

Testing Steps

# Step 1: Install eventlet (ensure version >= 0.37.0)
pip install eventlet>=0.37.0

# Step 2: Start worker with eventlet pool
python -m celery -A test_eventlet_shutdown worker --pool=eventlet --concurrency=4 --loglevel=info

# Step 3: Press Ctrl+C once worker is ready
# Expected (AFTER fix): Clean warm shutdown
# Expected (BEFORE fix): RuntimeError: do not call blocking functions from the mainloop

Backward Compatibility

✅ Fully backward compatible

No API changes
No configuration changes
Works with all eventlet versions (including those without the patch)
Works with gevent (same pattern applies)
Works without any concurrency pool (original os.write is used either way)

Related Issues

Fixes Celery warm shutdown with celery>=5.5.0b3 and eventlet>=0.37.0 #10083
Related to print is not thread-safe, so should not be used in signal handler #9222 (original change that introduced os.write in safe_say)

Checklist

I have verified that the fix works against the main branch of Celery.
I have added comments explaining why this change is necessary.
The fix is backward compatible.
No new dependencies are introduced.
I have added/updated tests (if applicable).
I have read the contribution guide.

codecov · 2026-02-15T10:41:44Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 87.53%. Comparing base (8ca3eae) to head (67f01e9).
⚠️ Report is 106 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main   #10123   +/-   ##
=======================================
  Coverage   87.52%   87.53%           
=======================================
  Files         153      153           
  Lines       19382    19383    +1     
  Branches     2229     2229           
=======================================
+ Hits        16965    16966    +1     
  Misses       2121     2121           
  Partials      296      296

Flag	Coverage Δ
unittests	`87.51% <100.00%> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copilot

Pull request overview

This PR fixes a RuntimeError that occurs during warm shutdown when using Celery with eventlet>=0.37.0. The issue stems from PR #9222, which changed safe_say() to use os.write() for signal-safe output, combined with eventlet 0.37.0's change to make os.write() use cooperative I/O. When a SIGINT signal handler runs inside eventlet's hub event loop and calls the patched os.write(), it triggers a RuntimeError.

Changes:

Saves the original os.write function before monkey-patching occurs
Updates safe_say() to use the saved original os.write instead of the potentially patched version
Adds comprehensive tests to verify the fix works correctly

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File	Description
celery/init.py	Saves `os.write` as `_original_os_write` before monkey-patching and exports it via `recreate_module()`
celery/apps/worker.py	Updates `safe_say()` to import and use `_original_os_write` instead of `os.write`, with explanatory docstring
t/unit/worker/test_worker.py	Adds tests to verify `_original_os_write` is properly saved and that `safe_say()` works even when `os.write` is patched

auvipy · 2026-02-15T13:27:38Z

btw is this backward compatible?

ChickenBenny · 2026-02-15T14:04:07Z

Yes, this fix is backward compatible~

No API changes - No public functions modified, no signatures changed
No config changes - No new settings, no breaking changes to existing behavior
Works across all scenarios:
- Eventlet < 0.37.0: ✅ (uses same os.write as before)
- Eventlet >= 0.37.0: ✅ (bypasses the problematic patch)
- Gevent: ✅ (identical behavior)
- Other pools (threads/prefork): ✅ (no impact)
- No concurrency pool: ✅ (works normally)

* fix: save the original os write before import eventlet * test: safe say use original os write --------- Co-authored-by: Asif Saif Uddin {"Auvi":"অভি"} <[email protected]>

This MR contains the following updates: | Package | Type | Update | Change | OpenSSF | |---|---|---|---|---| | [celery](https://docs.celeryq.dev/) ([source](https://github.com/celery/celery), [changelog](https://docs.celeryq.dev/en/stable/changelog.html)) | dependencies | patch | `5.6.2` → `5.6.3` | [![OpenSSF Scorecard](https://api.securityscorecards.dev/projects/github.com/celery/celery/badge)](https://securityscorecards.dev/viewer/?uri=github.com/celery/celery) | --- ### Release Notes <details> <summary>celery/celery (celery)</summary> ### [`v5.6.3`](https://github.com/celery/celery/releases/tag/v5.6.3) [Compare Source](celery/celery@v5.6.2...v5.6.3) #### What's Changed - Fix Django worker recursion bug + defensive checks for pool\_cls.**module** by [@maycuatroi1](https://github.com/maycuatroi1) in [#10048](celery/celery#10048) - Docs: Update user\_preload\_options example to use click. by [@jorsyk](https://github.com/jorsyk) in [#10056](celery/celery#10056) - Fix invalid configuration key "bootstrap\_servers" in Kafka demo by [@jorsyk](https://github.com/jorsyk) in [#10060](celery/celery#10060) - Fix broken images on PyPI page by [@Timour-Ilyas](https://github.com/Timour-Ilyas) in [#10066](celery/celery#10066) - Remove broken reference. by [@sueannioanis](https://github.com/sueannioanis) in [#10071](celery/celery#10071) - Removed --dist=loadscope from smoke tests by [@Nusnus](https://github.com/Nusnus) in [#10073](celery/celery#10073) - Docs: Clarify task\_retry signal args may be None by [@GangEunzzang](https://github.com/GangEunzzang) in [#10076](celery/celery#10076) - Update example for Django by [@sbc-khacnha](https://github.com/sbc-khacnha) in [#10081](celery/celery#10081) - Make tests compatible with pymongo >= 4.16 by [@cjwatson](https://github.com/cjwatson) in [#10074](celery/celery#10074) - fix: source install of cassandra-driver by [@Izzette](https://github.com/Izzette) in [#10105](celery/celery#10105) - fix: register task cross-reference role in Sphinx extension by [@veeceey](https://github.com/veeceey) in [#10100](celery/celery#10100) - fix: avoid cycle detection in native delayed delivery by [@Izzette](https://github.com/Izzette) in [#10095](celery/celery#10095) - fix(asynpool): avoid AttributeError when proc lacks \_sentinel\_poll by [@mriddle](https://github.com/mriddle) in [#10086](celery/celery#10086) - fix dusk\_astronomical horizon sign (+18 -> -18) by [@Mr-Neutr0n](https://github.com/Mr-Neutr0n) in [#10121](celery/celery#10121) - Fix/10106 onupdate col use lambda func by [@ChickenBenny](https://github.com/ChickenBenny) in [#10108](celery/celery#10108) - Fix warm shutdown RuntimeError with eventlet>=0.37.0 ([#10083](celery/celery#10083)) by [@ChickenBenny](https://github.com/ChickenBenny) in [#10123](celery/celery#10123) - Fix 10109 db backend connection health by [@ChickenBenny](https://github.com/ChickenBenny) in [#10124](celery/celery#10124) - Database Backend filter unsupport sql engine arguments with nullpool [#7355](celery/celery#7355) by [@ChickenBenny](https://github.com/ChickenBenny) in [#10134](celery/celery#10134) - fix(beat): correct argument order in Service.**reduce** by [@bysiber](https://github.com/bysiber) in [#10137](celery/celery#10137) - ci: declare explicit read-only token permissions in workflow jobs by [@Rohan5commit](https://github.com/Rohan5commit) in [#10139](celery/celery#10139) - chore: 'boto3to' to 'boto3 to' by [@cuiweixie](https://github.com/cuiweixie) in [#10133](celery/celery#10133) - Database Backend: Add missing index on date\_done (Fixes [#10097](celery/celery#10097)) by [@ChickenBenny](https://github.com/ChickenBenny) in [#10098](celery/celery#10098) - docs: fix typo in CONTRIBUTING.rst by [@Rohan5commit](https://github.com/Rohan5commit) in [#10141](celery/celery#10141) - Refer to Flower / Prometheus for monitoring by [@WilliamDEdwards](https://github.com/WilliamDEdwards) in [#10140](celery/celery#10140) - docs: remove duplicated words in broker and routing docs by [@Rohan5commit](https://github.com/Rohan5commit) in [#10146](celery/celery#10146) - docs: fix stale version reference and grammar in README by [@kelsonbrito50](https://github.com/kelsonbrito50) in [#10145](celery/celery#10145) - docs: fix wording in Celery 5.3 worker pool notes by [@Rohan5commit](https://github.com/Rohan5commit) in [#10149](celery/celery#10149) - docs: fix duplicated wording in 3.1 changelog entry by [@Rohan5commit](https://github.com/Rohan5commit) in [#10152](celery/celery#10152) - docs: fix changelog typo in context manager wording by [@Rohan5commit](https://github.com/Rohan5commit) in [#10144](celery/celery#10144) - Fix/10096 worker fails to reconnect after redis failover by [@ChickenBenny](https://github.com/ChickenBenny) in [#10151](celery/celery#10151) - Improve on\_after\_finalize signal documentation by [@Br1an67](https://github.com/Br1an67) in [#10155](celery/celery#10155) - Add non-commutative example to clarify partial arg ordering in canvas docs by [@Br1an67](https://github.com/Br1an67) in [#10157](celery/celery#10157) - Remove redundant test\_isa\_mapping test (fixes [#10077](celery/celery#10077)) by [@daniel7an](https://github.com/daniel7an) in [#10103](celery/celery#10103) - Upgrade pytest-celery to >=1.3.0 and adopt PYTEST\_CELERY\_PKG build arg by [@Nusnus](https://github.com/Nusnus) in [#10162](celery/celery#10162) - Remove deprecated args from redis get\_connection call by [@JaeHyuckSa](https://github.com/JaeHyuckSa) in [#10036](celery/celery#10036) - Fix [#6912](celery/celery#6912) rpc backend reconnection error by [@ChickenBenny](https://github.com/ChickenBenny) in [#10179](celery/celery#10179) - Fix NameError with TYPE\_CHECKING annotations on Python 3.14+ (PEP 649) by [@drichardson](https://github.com/drichardson) in [#10165](celery/celery#10165) - docs: Add elaboration on prefetch multiplier settings (worker\_prefetch\_multiplier) and worker\_eta\_task\_limit by [@tsangwailam](https://github.com/tsangwailam) in [#10181](celery/celery#10181) - Fix O(K²) message bloat in a chain of chords by [@Borzik](https://github.com/Borzik) in [#10171](celery/celery#10171) - Fix mock connection interfaces to prevent `TypeError` during exception handling by [@ChickenBenny](https://github.com/ChickenBenny) in [#10178](celery/celery#10178) - fix(trace): dispatch chain/callbacks on dedup fast-path for redelivered tasks by [@aurangzaib048](https://github.com/aurangzaib048) in [#10159](celery/celery#10159) - Extract `reconnect_on_error` to `BaseResultConsumer` by [@ChickenBenny](https://github.com/ChickenBenny) in [#10189](celery/celery#10189) - pep 649 by [@ericbuehl](https://github.com/ericbuehl) in [#10187](celery/celery#10187) - [Fix#9722](https://github.com/Fix/celery/issues/9722) friendly status errors for CLI by [@ChickenBenny](https://github.com/ChickenBenny) in [#10190](celery/celery#10190) - docs: clarify after\_return behavior for retried tasks by [@KianAnbarestani](https://github.com/KianAnbarestani) in [#10192](celery/celery#10192) - Add compression header to message protocol docs by [@Br1an67](https://github.com/Br1an67) in [#10156](celery/celery#10156) - docs: fix duplicated word in bootsteps comment by [@Rohan5commit](https://github.com/Rohan5commit) in [#10153](celery/celery#10153) - Remove outdated autoreloader section from extending docs by [@Br1an67](https://github.com/Br1an67) in [#10154](celery/celery#10154) - Fix: prioritize request ignore\_result over task definition by [@patri27826](https://github.com/patri27826) in [#10184](celery/celery#10184) - fix: clear the timer while catch the exception by [@ChickenBenny](https://github.com/ChickenBenny) in [#10218](celery/celery#10218) - Prepare for release: v5.6.3 by [@Nusnus](https://github.com/Nusnus) in [#10221](celery/celery#10221) #### New Contributors - [@maycuatroi1](https://github.com/maycuatroi1) made their first contribution in [#10048](celery/celery#10048) - [@jorsyk](https://github.com/jorsyk) made their first contribution in [#10056](celery/celery#10056) - [@Timour-Ilyas](https://github.com/Timour-Ilyas) made their first contribution in [#10066](celery/celery#10066) - [@sueannioanis](https://github.com/sueannioanis) made their first contribution in [#10071](celery/celery#10071) - [@GangEunzzang](https://github.com/GangEunzzang) made their first contribution in [#10076](celery/celery#10076) - [@sbc-khacnha](https://github.com/sbc-khacnha) made their first contribution in [#10081](celery/celery#10081) - [@veeceey](https://github.com/veeceey) made their first contribution in [#10100](celery/celery#10100) - [@mriddle](https://github.com/mriddle) made their first contribution in [#10086](celery/celery#10086) - [@Mr-Neutr0n](https://github.com/Mr-Neutr0n) made their first contribution in [#10121](celery/celery#10121) - [@ChickenBenny](https://github.com/ChickenBenny) made their first contribution in [#10108](celery/celery#10108) - [@bysiber](https://github.com/bysiber) made their first contribution in [#10137](celery/celery#10137) - [@Rohan5commit](https://github.com/Rohan5commit) made their first contribution in [#10139](celery/celery#10139) - [@cuiweixie](https://github.com/cuiweixie) made their first contribution in [#10133](celery/celery#10133) - [@kelsonbrito50](https://github.com/kelsonbrito50) made their first contribution in [#10145](celery/celery#10145) - [@Br1an67](https://github.com/Br1an67) made their first contribution in [#10155](celery/celery#10155) - [@daniel7an](https://github.com/daniel7an) made their first contribution in [#10103](celery/celery#10103) - [@drichardson](https://github.com/drichardson) made their first contribution in [#10165](celery/celery#10165) - [@tsangwailam](https://github.com/tsangwailam) made their first contribution in [#10181](celery/celery#10181) - [@Borzik](https://github.com/Borzik) made their first contribution in [#10171](celery/celery#10171) - [@aurangzaib048](https://github.com/aurangzaib048) made their first contribution in [#10159](celery/celery#10159) - [@ericbuehl](https://github.com/ericbuehl) made their first contribution in [#10187](celery/celery#10187) - [@KianAnbarestani](https://github.com/KianAnbarestani) made their first contribution in [#10192](celery/celery#10192) - [@patri27826](https://github.com/patri27826) made their first contribution in [#10184](celery/celery#10184) **Full Changelog**: <celery/celery@v5.6.2...v5.6.3> </details> --- - [ ] If you want to rebase/retry this MR, check this box --- This MR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).  See merge request swiss-armed-forces/cyber-command/cea/loom!419

bennyhsiao and others added 3 commits February 15, 2026 12:33

fix: save the original os write before import eventlet

8ea3d7d

test: safe say use original os write

7ceeff7

Merge branch 'main' into fix/10083-eventlet-warm-shutdown-crash

67f01e9

auvipy requested review from auvipy and Copilot February 15, 2026 10:45

Copilot started reviewing on behalf of auvipy February 15, 2026 10:45 View session

Copilot AI reviewed Feb 15, 2026

View reviewed changes

auvipy approved these changes Feb 15, 2026

View reviewed changes

auvipy added this to the 5.7.0 milestone Feb 15, 2026

auvipy merged commit 67b328a into celery:main Feb 15, 2026
326 checks passed

auvipy added the PR Type: Bugfix label Feb 15, 2026

mitchh456 mentioned this pull request Mar 31, 2026

Celery 5.6.3 removes deprecated redis get_connection args — verify Scout instrumentation scoutapp/scout_apm_python#837

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix warm shutdown RuntimeError with eventlet>=0.37.0 (#10083)#10123

Fix warm shutdown RuntimeError with eventlet>=0.37.0 (#10083)#10123
auvipy merged 3 commits into
celery:mainfrom
ChickenBenny:fix/10083-eventlet-warm-shutdown-crash

ChickenBenny commented Feb 15, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Feb 15, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

auvipy commented Feb 15, 2026

Uh oh!

ChickenBenny commented Feb 15, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

ChickenBenny commented Feb 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Problem

Root Cause

Call Stack

Solution

Why This Works

Changes

Testing

Manual Testing

Reproduction Script

Testing Steps

Backward Compatibility

Related Issues

Checklist

Uh oh!

codecov Bot commented Feb 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

auvipy commented Feb 15, 2026

Uh oh!

ChickenBenny commented Feb 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ChickenBenny commented Feb 15, 2026 •

edited

Loading

codecov Bot commented Feb 15, 2026 •

edited

Loading

ChickenBenny commented Feb 15, 2026 •

edited

Loading