[BUG] test_parser_and_batch_solver intermittently hangs inside LP BatchSolve on CUDA 13.3

[ramakrishnap-nv]

Describe the bug

test_lp_solver.py::test_parser_and_batch_solver intermittently hangs inside LP BatchSolve and never returns, taking the whole pytest cuopt step down when its time limit fires.

Evidence

Captured with faulthandler_timeout on a real CI runner (job 97564741474, conda-python-tests / 13.3.0, 3.14, arm64, ubuntu26.04, l4):

Timeout (0:10:00)!
Thread ... (most recent call first):
  File ".../cuopt/linear_programming/solver/solver.py", line 203 in BatchSolve
  File ".../cuopt/utilities/exception_handler.py", line 24 in func
  File ".../tests/linear_programming/test_lp_solver.py", line 595 in test_parser_and_batch_solver

solver.py:203 is return solver_wrapper.BatchSolve(...), so the block is in C++, not Python.

Frequency and scope

Seen on 13.3.0 only, across both amd64 and arm64:

config	observed
13.3.0, 3.13, amd64, rtxpro6000	hung 3 of 4 runs
13.3.0, 3.14, arm64, l4	hung
13.3.0, 3.14, amd64, h100	hung once, passed on retry
13.0.3, 12.x (all)	not seen

It is a full stop, not slowness: a healthy run of the suite completes in ~104s, while a hung one is still at 140/142 when killed. Raising the step limit from 30m to 45m did not change the outcome.

It is not specific to any PR -- it reproduces on a branch that changes only ci/*.sh.

What is ruled out

The test sets CUOPT_METHOD = PDLP, and solve_lp_with_method only dispatches to run_concurrent when the method is Concurrent. So the nested omp taskgroup / omp task / nested omp parallel constructs in solve.cu:1595-1690 are not on this path -- they all live inside run_concurrent. run_pdlp contains no OpenMP.

What is not yet established

The batch driver parallelises across models with #pragma omp parallel for num_threads(max_thread) (cpp/src/pdlp/utilities/cython_solve.cu:289, max_thread capped at 4), so several run_pdlp calls execute concurrently on one GPU. The blocking frame within that has not been identified -- the faulthandler stack proves Python is blocked in the Cython call but not where C++ stops.

One asymmetry worth checking when someone picks this up, flagged but unverified (cpp/src/pdlp/pdhg.cu:101-102):

graph_all{stream_view_, is_legacy_batch_mode || batch_mode_},
graph_prim_proj_gradient_dual{stream_view_, is_legacy_batch_mode},   // no batch_mode_

graph_all bypasses CUDA graphs in batch mode; graph_prim_proj_gradient_dual still captures. Capture uses cudaStreamCaptureModeThreadLocal (manual_cuda_graph.cuh:82), which is the multi-thread-safe mode, so this is a lead rather than a cause.

A native backtrace (gdb -p on a hung process) is the next step.

Note on severity

LP BatchSolve is deprecated (solver.py:193-199, "will be removed in a future release"), but it still ships, so callers can hit this today.

Test status

The test is being skipped to unblock CI. xfail does not work here: the test never returns, so pytest cannot observe a failure -- the outer timeout kills the whole step instead.