Repository navigation
Open
Description
In trace julia-9 from https://buildkite.com/organizations/julialang/pipelines/julia-master/builds/13170/jobs/01819131-f9a7-48e4-926f-23e48419c663/artifacts/018191c4-b7b8-461b-97c8-6f6f44bfc49f, we have when attempting to replay:
[FATAL src/ReplaySession.cc:632:enter_syscall()]
(task 14658 (rec:1978) at time 2968298)
-> Assertion `false' failed to hold. Expected syscall_bp_vm to be clear but it's 1978's address space with a breakpoint at 0x7ff02dca8744 while we're at 0x70000002
Tail of trace dump:
[snip]
real_time:52858533.422724 global_time:2968294, event:`SYSCALL: epoll_wait' (state:EXITING_SYSCALL) tid:579, ticks:415456293513
rax:0x1 rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x400 rsi:0x7f50b86166c0 rdi:0x5 rbp:0x0 rsp:0x681ffde0 r8:0x0 r9:0x7f50ccc3fca0 r10:0xbb9 r11:0x246 r12:0xbb9 r13:0x7f50d858006e r14:0x7f50b86166c0 r15:0x7f50d858306e rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xe8 fs_base:0x7f50df703180 gs_base:0x0
{ tid:579, addr:0x7f50b86166c0, length:0x3000 }
}
{
real_time:52858533.423096 global_time:2968295, event:`SYSCALLBUF_FLUSH' tid:579, ticks:415456303524
{ syscall:'clock_gettime', ret:0x0, size:0x20 }
{ syscall:'read', ret:0xfff, size:0x100f, desched:1 }
{ syscall:'clock_gettime', ret:0x0, size:0x20 }
{ syscall:'clock_gettime', ret:0x0, size:0x20 }
{ syscall:'epoll_wait', ret:0x1, size:0x1c }
{ syscall:'clock_gettime', ret:0x0, size:0x20 }
{ syscall:'read', ret:0x6e4, size:0x6f4, desched:1 }
{ syscall:'clock_gettime', ret:0x0, size:0x20 }
{ syscall:'epoll_wait', ret:0x0, size:0x10 }
}
{
real_time:52858533.423110 global_time:2968296, event:`SYSCALL: epoll_wait' (state:ENTERING_SYSCALL) tid:579, ticks:415456303524
rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x400 rsi:0x7f50bcb6de60 rdi:0x5 rbp:0x0 rsp:0x681ffde0 r8:0x0 r9:0x7f50ccc3fca0 r10:0xa36 r11:0x246 r12:0xa36 r13:0x7f50d85817f6 r14:0x7f50bcb6de60 r15:0x7f50d85847f6 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xe8 fs_base:0x7f50df703180 gs_base:0x0
}
{
real_time:52858533.423128 global_time:2968297, event:`SYSCALLBUF_RESET' tid:579, ticks:415456303524
}
{
real_time:52858533.423177 global_time:2968298, event:`SYSCALL: exit_group' (state:ENTERING_SYSCALL) tid:1978, ticks:372899180
rax:0xffffffffffffffda rbx:0x7ff02dd98760 rcx:0xffffffffffffffff rdx:0x8f rsi:0x3c rdi:0x8f rbp:0x8f rsp:0x7ff02157ef68 r8:0xe7 r9:0xffffffffffffff78 r10:0xfffffffffffff81f r11:0x246 r12:0x7ff02dd98760 r13:0x63d r14:0x7ff02dda1428 r15:0x0 rip:0x7ff02dca8746 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xe7 fs_base:0x7ff02dbbf180 gs_base:0x0
}
{
real_time:52858533.423188 global_time:2968299, event:`EXIT' tid:1978, ticks:372899180
}
Will look into this further.
Metadata
Metadata
Assignees
Labels
No labels
Activity
Keno commentedon Jul 5, 2022
There isn't really anything that prevents the tracee from just capturing the state arbitrarily and jumping to it later at arbitrary points (and real applications rely on this). From the kernel's perspective, getting a signal is a setjmp and
sigreturnis a longjmp with some slightly fancy edge cases, but we can't really rely on any sort of pairing.yuyichao commentedon Jul 5, 2022
I was mainly thinking that I'm pretty sure in an earlier version of the julia signal handler (may or may not be in the merged version) I was directly using assembly code to jump to a different target (also I think the SIGFPE handler still does that).
yuyichao commentedon Jul 5, 2022
yeah, so I was mainly wondering if we could notice that on the next sigreturn that the previous one didn't run. But I guess there's nothing illegal for the application to just spend hours in a signal handler before doing a normal sigreturn and short of reliably unwinding the stack we wouldn't know if that has happened.
I was also thinking the value of the stack pointer could be an indication that the tracee has abandoned the signal frame but if the application does any sort of tricks with the stack then that won't work....
Keno commentedon Jul 5, 2022
There's probably heuristics that could work reasonably well, but 99% probably isn't good enough here. So far I'm liking the patching and unpatching the best. There's some annoying edge cases, but it gives us the nice property that application signal handlers will never see any ip outside the application code. I think it'd be too expensive for every desched, but as long as it's only on delivered signals, I think we're probably ok.
Keno commentedon Jul 5, 2022
Actually, maybe we don't even have to unpatch at all. What if we just fudge the ip in the signal frame to point into the original location where the syscall instruction would have been and fix that back up on sigreturns. The signal handler would be able to unwind correctly, because the cfi still matches what was there before. Sure, if it actually went and read instruction memory, it'd get confused, but that's the case with my unpatching scheme also.
rocallahan commentedon Jul 7, 2022
Do you mean unpatch on sigreturns, or adjust the IP on sigreturns? I think the former would be safer right?
Keno commentedon Jul 7, 2022
I mean adjust the IP to point into the tail of the extended jump patch. I've got this working now and it seems to be working well. Just fighting with GDB a bit to give somewhat reasonable unwind info. Will have a PR up soon.
rocallahan commentedon Jul 7, 2022
OK. I wonder how you handle the trailing instructions of the syscall hooks.
Keno commentedon Jul 7, 2022
Just copy them into the extended jump patch, so the patch looks like
Redesign syscallbuf to always unwind on interruption
Add test case for switching out of desched syscall
Redesign syscallbuf to always unwind on interruption
Add test case for switching out of desched syscall
Redesign syscallbuf to always unwind on interruption
Add test case for switching out of desched syscall
Redesign syscallbuf to always unwind on interruption
Add test case for switching out of desched syscall
cebtenzzre commentedon Sep 29, 2023
I built from the PR (after cherry-picking some commits for gcc 13 compatibility), and still get this crash:
Details
Redesign syscallbuf to always unwind on interruption
Add test case for switching out of desched syscall
Redesign syscallbuf to always unwind on interruption
Add test case for switching out of desched syscall
Redesign syscallbuf to always unwind on interruption
Add test case for switching out of desched syscall
Redesign syscallbuf to always unwind on interruption
Add test case for switching out of desched syscall
Redesign syscallbuf to always unwind on interruption
Add test case for switching out of desched syscall
Redesign syscallbuf to always unwind on interruption
Add test case for switching out of desched syscall