Optimise allocation patterns for recvmmsg - #11387
Open
NelsonVides wants to merge 3 commits into
Open
Conversation
essio_recvmmsg allocated (zeroed, and freed) the entire vlen*(bufSz+ctrlSz) + metadata scratch block, plus a second block for the result-term array, on every call. At vlen=64, bufSz=2048, ctrlSz=1024 that is ~197 KB of malloc+memzero+free to receive even one datagram. Keep one grow-to-fit block per scheduler thread via thread-specific data. A recvmmsg NIF call runs start-to-finish on a single scheduler thread and uses the block only within that call (received data is copied out into fresh binaries before returning), so no locking is needed and no two calls ever touch the same block concurrently. Retained memory is bounded by the number of scheduler threads that have run recvmmsg (normal + dirty-IO), independent of the number of sockets. The result-term array is carved out of the same block. Add recvmmsg_pool_reuse_udp4 to socket_SUITE, exercising pool reuse and growth across many calls with varying VLen/BufSz.
Now that the scratch block persists with a stable base pointer, the "stable" mmsghdr/iovec fields (buffer pointers, iov, msg_name) only change when the block moves or the per-slot dimensions (vlen, bufSz, ctrlSz) change. Do the full O(vlen) setup loop only on such a (re)layout; on a matching reuse, restore just the fields the kernel overwrites (msg_namelen, msg_controllen, msg_flags, msg_len) and the one post-processing overwrites (msg_control), and only for the leading slots the previous call actually used. A grow of the block only happens when total_sz (hence the dimensions) increased, so the layout check already covers it and no separate signal is needed. Also drop the per-call sys_memzero of the bufs/ctrls arrays: by keeping the post-processing index at 0 until the allocation loop, the empty/error paths free nothing and the success path only touches freshly allocated slots, so the arrays never need pre-zeroing. Together this makes the per-call cost O(messages received) rather than O(vlen).
The result loop allocated a full bufSz data binary and a full ctrlSz control binary for every datagram, then recv_create_bin realloced the data binary down to the payload length. For a data-only receiver (e.g. DNS) with the default 1024-byte control size and small payloads, that is a ~2 KB alloc + realloc-down plus a wasted ~1 KB control alloc per datagram -- the allocator was the largest remaining recv-side cost once the scratch pool removed the per-call churn. Since the payload and any cmsgs are copied out of the (reused) scratch block, allocate the output binaries at exactly the received sizes: msgLen for data and the actual control length for the control binary. The data binary is then handed off by recv_create_bin without a realloc, and a datagram with no ancillary data costs no control allocation at all. The result terms are identical; msg_control still points at the (now right-sized) control binary for cmsg decoding.
Contributor
CT Test Results 4 files 199 suites 1h 57m 3s ⏱️ For more details on these failures, see this check. Results for commit 06eb668. ♻️ This comment has been updated with latest results. To speed up review, make sure that you have read Contributing to Erlang/OTP and that all checks pass. See the TESTING and DEVELOPMENT HowTo guides for details about how to run test locally. Artifacts// Erlang/OTP Github Action Bot |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reduces per-call allocation on the socket recvmmsg path. These optimise the case when batches aren't saturated, that is, when we request 64 datagrams but actually receive 3 or 4, we skip preparing the work for the requested 64 datagrams as it is currently done and prepare only for the 3-4 received. In my load-tests saturated batches are so much faster than
inetbut when barely loaded,socketfalls long behind.Three commits:
O(messages)per-call setup — the stable mmsghdr/iovec fields are set once per layout; a matching reuse only restores the kernel/post-processing-mutated fields for the slots the previous call used. Drops the per-call memzero too. Setup cost is nowO(messages received), notO(vlen).Testing: socket_SUITE recvmmsg cases pass, plus new recvmmsg_pool_reuse_udp4 (reuse/grow/varying-count) and recvmmsg_ctrl_udp4 (ip pktinfo control path — first non-data-only coverage). Each commit compiles standalone.
A benchmark on Linux 6.18, AMD Ryzen 9 9950X3D:
where
vis the requestedvLenandkthe actual number of datagrams.