Skip to content

Fix double-close of the CUDA-imported external-memory fd - #67

Merged
duburcqa merged 1 commit into
Genesis-Embodied-AI:mainfrom
duburcqa:fix-cuda-interop-double-close
Jul 23, 2026
Merged

Fix double-close of the CUDA-imported external-memory fd#67
duburcqa merged 1 commit into
Genesis-Embodied-AI:mainfrom
duburcqa:fix-cuda-interop-double-close

Conversation

@duburcqa

Copy link
Copy Markdown
Contributor

Problem

CudaImportedBuffer's destructor close()s the Vulkan-exported memory fd (ext_fd_). But that fd is imported into CUDA via cudaImportExternalMemory() with a cudaExternalMemoryHandleTypeOpaqueFd handle, which transfers ownership of the fd to the CUDA driver — the CUDA docs state that performing any operation on the fd after import is undefined behavior.

So the destructor does a double-close: cudaDestroyExternalMemory() lets the driver close the fd, and then we close() it again. Once that fd number has been recycled by an unrelated open(), the driver's teardown close (a raw syscall) lands on the wrong file, and the process starts seeing spurious OSError: [Errno 9] Bad file descriptor in completely unrelated code.

Symptom

In Genesis this manifested as batch-render tests crashing the quadrants JIT while it hashed a kernel's Python source — but only when multiple GPU tests share a process or run under pytest --forked (where fd churn makes the recycle likely). A single test in a fresh process was unaffected, which is why it looked like a fork/isolation problem rather than a plain double-close.

Fix

Drop the erroneous close(ext_fd_); CUDA owns the fd after import and releases it with the external memory.

Validation (cluster, RTX)

The two scenarios that failed deterministically with EBADF before now both pass:

Scenario before after
2 GPU batch tests in one process FAIL (EBADF) PASS
pytest --forked FAIL (EBADF) PASS

cudaImportExternalMemory() with a cudaExternalMemoryHandleTypeOpaqueFd handle
takes ownership of the file descriptor - the CUDA docs state any operation on it
afterwards is undefined behavior. CudaImportedBuffer's destructor nonetheless
close()d it on top of cudaDestroyExternalMemory(), a double-close: the driver
closes that fd during teardown, and once its number is recycled by an unrelated
open() the process hits spurious 'Bad file descriptor' errors. This surfaced as
batch-render tests crashing genesis's JIT source hashing when multiple GPU tests
share a process or run under pytest --forked. Let CUDA own the fd; drop the close().
@duburcqa
duburcqa merged commit 7705c48 into Genesis-Embodied-AI:main Jul 23, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant