Save all volatile argument registers in GC probe hijack frames on AMD64#126848
Open
Save all volatile argument registers in GC probe hijack frames on AMD64#126848
Conversation
… frames On AMD64, when the GC hijacks a thread via RhpGcProbeHijack, the hijack stub creates a PInvokeTransitionFrame. Previously on Windows, only RAX and RCX were saved, and on Unix only RAX, RCX, and RDX were saved. If the GC info for the managed frame at the hijack point reports other volatile registers (RDX, R8, R9) as live GC references, the StackFrameIterator would find NULL save locations and crash (AV in SVR::GCHeap::Promote). Windows changes: - FixupHijackedCallstack now uses R10/R11 for thread pointer instead of RDX/R8, preserving all volatile argument registers (RAX, RCX, RDX, R8, R9) - PUSH_PROBE_FRAME/POP_PROBE_FRAME extended to save/restore RDX, R8, R9 - RhpGcProbeHijack flags include PTFF_SAVE_RDX + PTFF_SAVE_R8 + PTFF_SAVE_R9 - AsmMacros.inc gains PTFF_SAVE_RDX, PTFF_SAVE_R8, PTFF_SAVE_R9 definitions Unix changes: - FixupHijackedCallstack now also saves/restores R8 and R9 across GETTHREAD - Uses R10 instead of R8 for scratch in hijack fixup and bitmask passing - PUSH_PROBE_FRAME/POP_PROBE_FRAME extended to save/restore R8, R9 - RhpGcProbeHijack flags include PTFF_SAVE_R8 + PTFF_SAVE_R9 - unixasmmacrosamd64.inc gains PTFF_SAVE_R8, PTFF_SAVE_R9 definitions Agent-Logs-Url: https://github.com/dotnet/runtime/sessions/411be3e7-43a1-4bc3-b896-e27d92fe37c3 Co-authored-by: mangod9 <61718172+mangod9@users.noreply.github.com>
Agent-Logs-Url: https://github.com/dotnet/runtime/sessions/411be3e7-43a1-4bc3-b896-e27d92fe37c3 Co-authored-by: mangod9 <61718172+mangod9@users.noreply.github.com>
Copilot
AI
changed the title
[WIP] Fix test failure for FastTailCallCandidates
Save all volatile argument registers (RDX, R8, R9) in GC probe hijack frames on AMD64
Apr 13, 2026
jkotas
reviewed
Apr 13, 2026
Contributor
|
Tagging subscribers to this area: @agocke, @dotnet/ilc-contrib |
On Unix System V ABI, RSI and RDI are volatile argument-passing registers (1st and 2nd integer args) that could contain live GC references at hijack points. On Windows, they are already saved as callee-saved registers in PTFF_SAVE_ALL_PRESERVED. Unix changes: - FixupHijackedCallstack now saves/restores RSI and RDI across GETTHREAD - PUSH_PROBE_FRAME saves RSI and RDI between R12 and RBX, matching the StackFrameIterator's flag processing order (RBX, RSI, RDI, R12...) - POP_PROBE_FRAME restores RSI and RDI in correct order - RhpGcProbeHijack flags include PTFF_SAVE_RSI + PTFF_SAVE_RDI - unixasmmacrosamd64.inc gains PTFF_SAVE_RSI and PTFF_SAVE_RDI definitions Agent-Logs-Url: https://github.com/dotnet/runtime/sessions/85ca9ec1-057b-4a3c-8182-e6da274d44b6 Co-authored-by: mangod9 <61718172+mangod9@users.noreply.github.com>
Copilot
AI
changed the title
Save all volatile argument registers (RDX, R8, R9) in GC probe hijack frames on AMD64
Save all volatile argument registers in GC probe hijack frames on AMD64
Apr 13, 2026
Member
|
/azp run runtime-nativeaot-outerloop |
|
Azure Pipelines successfully started running 1 pipeline(s). |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR fixes NativeAOT AMD64 GC hijack/probe frames so that volatile argument registers that may hold live GC references at hijack points are preserved and correctly described to StackFrameIterator, preventing AVs observed in CrossGen2 during GC promotion.
Changes:
- Extend GC probe transition frame save/restore logic to include additional volatile argument registers (Windows: RDX/R8/R9; Unix: RSI/RDI/R8/R9 plus RDX) and update the corresponding hijack save flags.
- Adjust
FixupHijackedCallstackscratch register usage to avoid clobbering argument registers (Windows: use R10/R11; Unix: preserve args acrossINLINE_GETTHREAD). - Add/update PTFF save-flag definitions in AMD64 asm macro include files for both Windows and Unix.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| src/coreclr/nativeaot/Runtime/unix/unixasmmacrosamd64.inc | Adds PTFF_SAVE_* definitions for additional registers used by Unix AMD64 probe/hijack frames. |
| src/coreclr/nativeaot/Runtime/amd64/GcProbe.asm | Windows AMD64: saves/restores RDX/R8/R9 in probe frames; avoids clobber in hijack fixup by using R10/R11; updates save-flag mask. |
| src/coreclr/nativeaot/Runtime/amd64/GcProbe.S | Unix AMD64: saves/restores RSI/RDI/R8/R9 (and updates flag mask/order) and avoids clobber across INLINE_GETTHREAD. |
| src/coreclr/nativeaot/Runtime/amd64/AsmMacros.inc | Adds PTFF_SAVE_RDX/R8/R9 constants to match PInvokeTransitionFrameFlags. |
Comment on lines
+100
to
+104
| // preserve RSI, RDI, R8 and R9 as they may contain GC refs | ||
| push rsi | ||
| push rdi | ||
| push r8 | ||
| push r9 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
CrossGen2 crashes with AV in
SVR::GCHeap::Promotewhen the GC hijacks a thread at a point where volatile argument registers contain live GC references. The hijack stub'sPInvokeTransitionFramedidn't save these registers, soStackFrameIteratorfound NULL save locations when GC info reported them as live.Windows AMD64:
FixupHijackedCallstackused RDX for the thread pointer and R8 as scratch, destroying their original values. Only RAX/RCX were saved.Unix AMD64: RSI, RDI, R8, and R9 were destroyed by the
INLINE_GETTHREADfunction call and never saved. On the System V ABI, RSI and RDI are volatile argument-passing registers (1st and 2nd integer args) that could contain live GC references at hijack points. On Windows, RSI/RDI are already covered as callee-saved registers inPTFF_SAVE_ALL_PRESERVED.Changes
FixupHijackedCallstack(Windows): Use R10/R11 instead of RDX/R8 for thread pointer and scratch, preserving all volatile argument registersFixupHijackedCallstack(Unix): Save/restore RSI, RDI, R8, and R9 aroundINLINE_GETTHREAD; use R10 instead of R8 for scratchPUSH/POP_PROBE_FRAME(Windows): Save/restore RDX, R8, R9 in correctm_PreservedRegsorder matchingStackFrameIteratorflag processingPUSH/POP_PROBE_FRAME(Unix): Save/restore RSI, RDI, RDX, R8, R9 in correctm_PreservedRegsorder matchingStackFrameIteratorflag processing (RSI/RDI placed between R12 and RBX)PTFF_SAVE_RDX + PTFF_SAVE_R8 + PTFF_SAVE_R9PTFF_SAVE_RSI + PTFF_SAVE_RDI + PTFF_SAVE_RDX + PTFF_SAVE_R8 + PTFF_SAVE_R9PTFF_SAVE_RDX,PTFF_SAVE_R8,PTFF_SAVE_R9toAsmMacros.inc; addPTFF_SAVE_RSI,PTFF_SAVE_RDI,PTFF_SAVE_R8,PTFF_SAVE_R9tounixasmmacrosamd64.inc