Skip to content

fix: gracefully stop uvicorn threads during shutdown#687

Open
mkg0908 wants to merge 1 commit into
Lightning-AI:mainfrom
mkg0908:fix-684-thread-shutdown
Open

fix: gracefully stop uvicorn threads during shutdown#687
mkg0908 wants to merge 1 commit into
Lightning-AI:mainfrom
mkg0908:fix-684-thread-shutdown

Conversation

@mkg0908

@mkg0908 mkg0908 commented Jun 2, 2026

Copy link
Copy Markdown

Summary

  • retain the Uvicorn server instance for thread-backed API workers
  • request thread shutdown through server.should_exit instead of calling process-only terminate() and kill() methods
  • add regression coverage for graceful thread shutdown

Tests

  • .venv/bin/python -m ruff check src/litserve/server.py tests/unit/test_lit_server.py
  • .venv/bin/python -m compileall -q src/litserve/server.py tests/unit/test_lit_server.py
  • focused shutdown regression check using a thread-backed Uvicorn server double

Closes #684

@codecov-commenter

codecov-commenter commented Jun 3, 2026

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 85.71429% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 85%. Comparing base (a69b635) to head (dd4e6d5).
⚠️ Report is 4 commits behind head on main.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@         Coverage Diff         @@
##           main   #687   +/-   ##
===================================
- Coverage    85%    85%   -0%     
===================================
  Files        39     39           
  Lines      3282   3289    +7     
===================================
+ Hits       2778   2782    +4     
- Misses      504    507    +3     
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Comment thread src/litserve/server.py
Comment on lines 1583 to +1586
w = threading.Thread(
target=server.run, args=(response_queue_id, sockets), name=f"LitServer-{response_queue_id}"
)
setattr(w, "_litserve_server", server)

@bhimrazy bhimrazy Jun 3, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The setattr/getattr approach works but I think a small subclass makes the intent clearer and the handle typed:

class _UvicornServerThread(threading.Thread):
    def __init__(self, server: _Server, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.server = server
w = _UvicornServerThread(
    server, target=server.run, args=(response_queue_id, sockets), name=f"LitServer-{response_queue_id}"
)

wdyt ?

cc: @andyland

@racinmat

racinmat commented Jun 6, 2026

Copy link
Copy Markdown

Unfortunately this PR doesn't fix the issue. I just tried it and it was hanging, I got this output.

C:\Projects\others\LitServe\.venv\Scripts\python.exe -X pycache_prefix=C:\Users\Matej.Racinsky\AppData\Local\JetBrains\PyCharm2026.1\cpython-cache C:/Users/Matej.Racinsky/AppData/Local/Programs/PyCharm/plugins/python-ce/helpers/pydev/pydevd.py --multiprocess --qt-support=auto --client 127.0.0.1 --port 19612 --file C:\Projects\others\LitServe\litserve_win_shutdown2.py 
import sys; print('Python %s on %s' % (sys.version, sys.platform))
Connected to pydev debugger (build 261.24374.152)
[MAIN] Starting LitServer...
INFO:     Uvicorn running on http://0.0.0.0:8765 (Press CTRL+C to quit)
C:\Projects\others\LitServe\src\litserve\server.py:1495: UserWarning: Windows does not support forking. Using threads api_server_worker_type will be set to 'thread'
  warnings.warn(
2026-06-06 16:26:11,772 - MainProcess[32988] - litserve.server - DEBUG - server.py:1164 - One or more workers are ready to serve requests
Swagger UI is available at http://0.0.0.0:8765/docs
INFO:     Started server process [32988]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
2026-06-06 16:26:19,972 - MainProcess[32988] - litserve.server - INFO - server.py:1527 - KeyboardInterrupt received. Initiating graceful shutdown.
2026-06-06 16:26:19,972 - MainProcess[32988] - litserve.server - INFO - server.py:1190 - Shutting down LitServe...
2026-06-06 16:26:19,973 - MainProcess[32988] - litserve.server - DEBUG - server.py:1194 - KeyboardInterrupt detected - skipping transport cleanup to avoid hanging

and it never finished.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Threads not properly killed on windows

4 participants