Skip to content

Replay failed with "Expected syscall_bp_vm to be clear" #3285

Open

Description

@Keno

In trace julia-9 from https://buildkite.com/organizations/julialang/pipelines/julia-master/builds/13170/jobs/01819131-f9a7-48e4-926f-23e48419c663/artifacts/018191c4-b7b8-461b-97c8-6f6f44bfc49f, we have when attempting to replay:

[FATAL src/ReplaySession.cc:632:enter_syscall()]
 (task 14658 (rec:1978) at time 2968298)
 -> Assertion `false' failed to hold. Expected syscall_bp_vm to be clear but it's 1978's address space with a breakpoint at 0x7ff02dca8744 while we're at 0x70000002
Tail of trace dump:

[snip]

  real_time:52858533.422724 global_time:2968294, event:`SYSCALL: epoll_wait' (state:EXITING_SYSCALL) tid:579, ticks:415456293513
rax:0x1 rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x400 rsi:0x7f50b86166c0 rdi:0x5 rbp:0x0 rsp:0x681ffde0 r8:0x0 r9:0x7f50ccc3fca0 r10:0xbb9 r11:0x246 r12:0xbb9 r13:0x7f50d858006e r14:0x7f50b86166c0 r15:0x7f50d858306e rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xe8 fs_base:0x7f50df703180 gs_base:0x0
  { tid:579, addr:0x7f50b86166c0, length:0x3000 }
}
{
  real_time:52858533.423096 global_time:2968295, event:`SYSCALLBUF_FLUSH' tid:579, ticks:415456303524
  { syscall:'clock_gettime', ret:0x0, size:0x20 }
  { syscall:'read', ret:0xfff, size:0x100f, desched:1 }
  { syscall:'clock_gettime', ret:0x0, size:0x20 }
  { syscall:'clock_gettime', ret:0x0, size:0x20 }
  { syscall:'epoll_wait', ret:0x1, size:0x1c }
  { syscall:'clock_gettime', ret:0x0, size:0x20 }
  { syscall:'read', ret:0x6e4, size:0x6f4, desched:1 }
  { syscall:'clock_gettime', ret:0x0, size:0x20 }
  { syscall:'epoll_wait', ret:0x0, size:0x10 }
}
{
  real_time:52858533.423110 global_time:2968296, event:`SYSCALL: epoll_wait' (state:ENTERING_SYSCALL) tid:579, ticks:415456303524
rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x400 rsi:0x7f50bcb6de60 rdi:0x5 rbp:0x0 rsp:0x681ffde0 r8:0x0 r9:0x7f50ccc3fca0 r10:0xa36 r11:0x246 r12:0xa36 r13:0x7f50d85817f6 r14:0x7f50bcb6de60 r15:0x7f50d85847f6 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xe8 fs_base:0x7f50df703180 gs_base:0x0
}
{
  real_time:52858533.423128 global_time:2968297, event:`SYSCALLBUF_RESET' tid:579, ticks:415456303524
}
{
  real_time:52858533.423177 global_time:2968298, event:`SYSCALL: exit_group' (state:ENTERING_SYSCALL) tid:1978, ticks:372899180
rax:0xffffffffffffffda rbx:0x7ff02dd98760 rcx:0xffffffffffffffff rdx:0x8f rsi:0x3c rdi:0x8f rbp:0x8f rsp:0x7ff02157ef68 r8:0xe7 r9:0xffffffffffffff78 r10:0xfffffffffffff81f r11:0x246 r12:0x7ff02dd98760 r13:0x63d r14:0x7ff02dda1428 r15:0x0 rip:0x7ff02dca8746 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xe7 fs_base:0x7ff02dbbf180 gs_base:0x0
}
{
  real_time:52858533.423188 global_time:2968299, event:`EXIT' tid:1978, ticks:372899180
}

Will look into this further.

Activity

  1. Keno commented on Jul 5, 2022

    @Keno
    MemberAuthor

    Would RR know if the sig return finished without returning to where it started?

    There isn't really anything that prevents the tracee from just capturing the state arbitrarily and jumping to it later at arbitrary points (and real applications rely on this). From the kernel's perspective, getting a signal is a setjmp and sigreturn is a longjmp with some slightly fancy edge cases, but we can't really rely on any sort of pairing.

  2. yuyichao commented on Jul 5, 2022

    @yuyichao
    Contributor

    I was mainly thinking that I'm pretty sure in an earlier version of the julia signal handler (may or may not be in the merged version) I was directly using assembly code to jump to a different target (also I think the SIGFPE handler still does that).

  3. yuyichao commented on Jul 5, 2022

    @yuyichao
    Contributor

    but we can't really rely on any sort of pairing

    yeah, so I was mainly wondering if we could notice that on the next sigreturn that the previous one didn't run. But I guess there's nothing illegal for the application to just spend hours in a signal handler before doing a normal sigreturn and short of reliably unwinding the stack we wouldn't know if that has happened.

    I was also thinking the value of the stack pointer could be an indication that the tracee has abandoned the signal frame but if the application does any sort of tricks with the stack then that won't work....

  4. Keno commented on Jul 5, 2022

    @Keno
    MemberAuthor

    There's probably heuristics that could work reasonably well, but 99% probably isn't good enough here. So far I'm liking the patching and unpatching the best. There's some annoying edge cases, but it gives us the nice property that application signal handlers will never see any ip outside the application code. I think it'd be too expensive for every desched, but as long as it's only on delivered signals, I think we're probably ok.

  5. Keno commented on Jul 5, 2022

    @Keno
    MemberAuthor

    Actually, maybe we don't even have to unpatch at all. What if we just fudge the ip in the signal frame to point into the original location where the syscall instruction would have been and fix that back up on sigreturns. The signal handler would be able to unwind correctly, because the cfi still matches what was there before. Sure, if it actually went and read instruction memory, it'd get confused, but that's the case with my unpatching scheme also.

  6. rocallahan commented on Jul 7, 2022

    @rocallahan
    Collaborator

    fix that back up on sigreturns.

    Do you mean unpatch on sigreturns, or adjust the IP on sigreturns? I think the former would be safer right?

  7. Keno commented on Jul 7, 2022

    @Keno
    MemberAuthor

    I mean adjust the IP to point into the tail of the extended jump patch. I've got this working now and it seems to be working well. Just fighting with GDB a bit to give somewhat reasonable unwind info. Will have a PR up soon.

  8. rocallahan commented on Jul 7, 2022

    @rocallahan
    Collaborator

    OK. I wonder how you handle the trailing instructions of the syscall hooks.

  9. Keno commented on Jul 7, 2022

    @Keno
    MemberAuthor

    OK. I wonder how you handle the trailing instructions of the syscall hooks.

    Just copy them into the extended jump patch, so the patch looks like

    <setup instructions>
    callq syscall_hook
    # Bail path returns here with stack already switched back
    syscall
    <trailing instructions>
    # Normal path could return here (but in practice just jumps right through to make GDB happier)
    jmpq *return_addr
    
  10. added 6 commits that reference this issue on Jul 8, 2022
    7b9abcf
    74ae10a
    ab53550
    8971638
    1cf85b7
    7eb33ab
  11. added 2 commits that reference this issue on Jul 17, 2022
    fd2660c
    618715c
  12. cebtenzzre commented on Sep 29, 2023

    @cebtenzzre

    I built from the PR (after cherry-picking some commits for gcc 13 compatibility), and still get this crash:

    Details
    [FATAL /usr/src/debug/rr-git/rr/src/ReplaySession.cc:635:enter_syscall()]                                                                                                                     
     (task 32428 (rec:32354) at time 33968)
     -> Assertion `false' failed to hold. Expected syscall_bp_vm to be clear but it's 32354's address space with a breakpoint at 0x7f967e923059 while we're at 0x7f967f069122
    Tail of trace dump:
    {
      real_time:8147.767345 global_time:33948, event:`SYSCALL: openat' (state:ENTERING_SYSCALL) tid:32354, ticks:319645440
    rax:0xffffffffffffffda rbx:0x7f967cab128c rcx:0xffffffffffffffff rdx:0x90800 rsi:0x5617be879efd rdi:0xffffff9c rbp:0x7ffe0e33fcc0 rsp:0x7ffe0e33fc30 r8:0x5617bd4df010 r9:0x0 r10:0x0 r11:0x246 r12:0x5617be879efd r13:0x0 r14:0x5617be870850 r15:0x5617be879f1a rip:0x7f967f0690af eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x101 fs_base:0x7f967efd1e80 gs_base:0x0
    }
    {
      real_time:8147.767352 global_time:33949, event:`SYSCALLBUF_RESET' tid:32354, ticks:319645440
    }
    {
      real_time:8147.767387 global_time:33950, event:`SYSCALL: openat' (state:EXITING_SYSCALL) tid:32354, ticks:319645440
    rax:0xfffffffffffffffe rbx:0x7f967cab128c rcx:0xffffffffffffffff rdx:0x90800 rsi:0x5617be879efd rdi:0xffffff9c rbp:0x7ffe0e33fcc0 rsp:0x7ffe0e33fc30 r8:0x5617bd4df010 r9:0x0 r10:0x0 r11:0x246 r12:0x5617be879efd r13:0x0 r14:0x5617be870850 r15:0x5617be879f1a rip:0x7f967f0690af eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x101 fs_base:0x7f967efd1e80 gs_base:0x0
    }
    {
      real_time:8147.767441 global_time:33951, event:`SYSCALLBUF_FLUSH' tid:32354, ticks:319645502
      { syscall:'openat', ret:0xfffffffffffffffe, size:0x10, desched:1 }
    }
    {
      real_time:8147.767448 global_time:33952, event:`SYSCALL: openat' (state:ENTERING_SYSCALL) tid:32354, ticks:319645502
    rax:0xffffffffffffffda rbx:0x7f967cab128c rcx:0xffffffffffffffff rdx:0x90800 rsi:0x5617be879f1a rdi:0xffffff9c rbp:0x7ffe0e33fcc0 rsp:0x7ffe0e33fc30 r8:0x5617bd4df010 r9:0x0 r10:0x0 r11:0x246 r12:0x5617be879f1a r13:0x0 r14:0x5617be870850 r15:0x5617be879f4b rip:0x7f967f0690af eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x101 fs_base:0x7f967efd1e80 gs_base:0x0
    }
    {
      real_time:8147.767454 global_time:33953, event:`SYSCALLBUF_RESET' tid:32354, ticks:319645502
    }
    {
      real_time:8147.767491 global_time:33954, event:`SYSCALL: openat' (state:EXITING_SYSCALL) tid:32354, ticks:319645502
    rax:0xfffffffffffffffe rbx:0x7f967cab128c rcx:0xffffffffffffffff rdx:0x90800 rsi:0x5617be879f1a rdi:0xffffff9c rbp:0x7ffe0e33fcc0 rsp:0x7ffe0e33fc30 r8:0x5617bd4df010 r9:0x0 r10:0x0 r11:0x246 r12:0x5617be879f1a r13:0x0 r14:0x5617be870850 r15:0x5617be879f4b rip:0x7f967f0690af eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x101 fs_base:0x7f967efd1e80 gs_base:0x0
    }
    {
      real_time:8147.767538 global_time:33955, event:`SYSCALLBUF_FLUSH' tid:32354, ticks:319645562
      { syscall:'openat', ret:0xfffffffffffffffe, size:0x10, desched:1 }
    }
    {
      real_time:8147.767545 global_time:33956, event:`SYSCALL: openat' (state:ENTERING_SYSCALL) tid:32354, ticks:319645562
    rax:0xffffffffffffffda rbx:0x7f967cab128c rcx:0xffffffffffffffff rdx:0x90800 rsi:0x5617be879f4b rdi:0xffffff9c rbp:0x7ffe0e33fcc0 rsp:0x7ffe0e33fc30 r8:0x5617bd4df010 r9:0x0 r10:0x0 r11:0x246 r12:0x5617be879f4b r13:0x0 r14:0x5617be870850 r15:0x5617be879f74 rip:0x7f967f0690af eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x101 fs_base:0x7f967efd1e80 gs_base:0x0
    }
    {
      real_time:8147.767551 global_time:33957, event:`SYSCALLBUF_RESET' tid:32354, ticks:319645562
    }
    {
      real_time:8147.767587 global_time:33958, event:`SYSCALL: openat' (state:EXITING_SYSCALL) tid:32354, ticks:319645562
    rax:0xfffffffffffffffe rbx:0x7f967cab128c rcx:0xffffffffffffffff rdx:0x90800 rsi:0x5617be879f4b rdi:0xffffff9c rbp:0x7ffe0e33fcc0 rsp:0x7ffe0e33fc30 r8:0x5617bd4df010 r9:0x0 r10:0x0 r11:0x246 r12:0x5617be879f4b r13:0x0 r14:0x5617be870850 r15:0x5617be879f74 rip:0x7f967f0690af eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x101 fs_base:0x7f967efd1e80 gs_base:0x0
    }
    {
      real_time:8147.768371 global_time:33959, event:`SYSCALLBUF_FLUSH' tid:32354, ticks:319645858
      { syscall:'openat', ret:0x3, size:0x10, desched:1 }
      { syscall:'readlinkat', ret:0x22, size:0x32 }
    }
    {
      real_time:8147.768379 global_time:33960, event:`SYSCALL: fstatat' (state:ENTERING_SYSCALL) tid:32354, ticks:319645858
    rax:0xffffffffffffffda rbx:0x3 rcx:0xffffffffffffffff rdx:0x7ffe0e33fbe0 rsi:0x7f967e9b8bd5 rdi:0x3 rbp:0x7ffe0e33fbe0 rsp:0x7ffe0e33fbd8 r8:0x0 r9:0x0 r10:0x1000 r11:0x246 r12:0x5617be879f74 r13:0x0 r14:0x5617be870850 r15:0x5617be879f96 rip:0x7f967e91d64e eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x106 fs_base:0x7f967efd1e80 gs_base:0x0
    }
    {
      real_time:8147.768386 global_time:33961, event:`SYSCALLBUF_RESET' tid:32354, ticks:319645858
    }
    {
      real_time:8147.768421 global_time:33962, event:`SYSCALL: fstatat' (state:EXITING_SYSCALL) tid:32354, ticks:319645858
    rax:0x0 rbx:0x3 rcx:0xffffffffffffffff rdx:0x7ffe0e33fbe0 rsi:0x7f967e9b8bd5 rdi:0x3 rbp:0x7ffe0e33fbe0 rsp:0x7ffe0e33fbd8 r8:0x0 r9:0x0 r10:0x1000 r11:0x246 r12:0x5617be879f74 r13:0x0 r14:0x5617be870850 r15:0x5617be879f96 rip:0x7f967e91d64e eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x106 fs_base:0x7f967efd1e80 gs_base:0x0
      { tid:32354, addr:0x7ffe0e33fbe0, length:0x90 }
    }
    {
      real_time:8147.768512 global_time:33963, event:`SYSCALLBUF_FLUSH' tid:32354, ticks:319647896
      { syscall:'getdents64', ret:0x68, size:0x78 }
      { syscall:'access', ret:0x0, size:0x10 }
      { syscall:'access', ret:0x0, size:0x10 }
      { syscall:'access', ret:0x0, size:0x10 }
      { syscall:'getdents64', ret:0x0, size:0x10 }
      { syscall:'close', ret:0x0, size:0x10 }
      { syscall:'openat', ret:0x3, size:0x10, desched:1 }
      { syscall:'readlinkat', ret:0x42, size:0x52 }
    }
    {
      real_time:8147.768518 global_time:33964, event:`SYSCALL: fstatat' (state:ENTERING_SYSCALL) tid:32354, ticks:319647896
    rax:0xffffffffffffffda rbx:0x5617bdacb4d0 rcx:0xffffffffffffffff rdx:0x7ffe0e340ae0 rsi:0x7f967e9b8bd5 rdi:0x3 rbp:0x7f967e9f5070 rsp:0x7ffe0e340ad8 r8:0x8 r9:0x1 r10:0x1000 r11:0x246 r12:0x7ffe0e340c80 r13:0x100 r14:0x7ffe0e340c80 r15:0x7f967efd1e80 rip:0x7f967e91d64e eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x106 fs_base:0x7f967efd1e80 gs_base:0x0
    }
    {
      real_time:8147.768522 global_time:33965, event:`SYSCALLBUF_RESET' tid:32354, ticks:319647896
    }
    {
      real_time:8147.768545 global_time:33966, event:`SYSCALL: fstatat' (state:EXITING_SYSCALL) tid:32354, ticks:319647896
    rax:0x0 rbx:0x5617bdacb4d0 rcx:0xffffffffffffffff rdx:0x7ffe0e340ae0 rsi:0x7f967e9b8bd5 rdi:0x3 rbp:0x7f967e9f5070 rsp:0x7ffe0e340ad8 r8:0x8 r9:0x1 r10:0x1000 r11:0x246 r12:0x7ffe0e340c80 r13:0x100 r14:0x7ffe0e340c80 r15:0x7f967efd1e80 rip:0x7f967e91d64e eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x106 fs_base:0x7f967efd1e80 gs_base:0x0
      { tid:32354, addr:0x7ffe0e340ae0, length:0x90 }
    }
    {
      real_time:8147.770177 global_time:33967, event:`SYSCALLBUF_FLUSH' tid:32354, ticks:319767963
      { syscall:'read', ret:0x1000, size:0x1010, desched:1 }
      { syscall:'read', ret:0x1000, size:0x1010, desched:1 }
      { syscall:'read', ret:0x1000, size:0x1010, desched:1 }
      { syscall:'read', ret:0x1000, size:0x1010, desched:1 }
      { syscall:'read', ret:0x1000, size:0x1010, desched:1 }
      { syscall:'read', ret:0x1000, size:0x1010, desched:1 }
      { syscall:'read', ret:0x1000, size:0x1010, desched:1 }
      { syscall:'read', ret:0x1000, size:0x1010, desched:1 }
      { syscall:'read', ret:0x1000, size:0x1010, desched:1 }
      { syscall:'read', ret:0x1000, size:0x1010, desched:1 }
      { syscall:'read', ret:0x1000, size:0x1010, desched:1 }
      { syscall:'read', ret:0x1000, size:0x1010, desched:1 }
      { syscall:'read', ret:0x1000, size:0x1010, desched:1 }
      { syscall:'read', ret:0x1000, size:0x1010, desched:1 }
      { syscall:'read', ret:0x1000, size:0x1010, desched:1 }
      { syscall:'read', ret:0x1000, size:0x1010, desched:1 }
      { syscall:'read', ret:0x1000, size:0x1010, desched:1 }
      { syscall:'read', ret:0x59a, size:0x5aa, desched:1 }
      { syscall:'read', ret:0x0, size:0x10, desched:1 }
      { syscall:'lseek', ret:0x1159a, size:0x10 }
      { syscall:'lseek', ret:0x0, size:0x10 }
      { syscall:'lseek', ret:0x0, size:0x10 }
      { syscall:'ioctl', ret:0x0, size:0x10 }
      { syscall:'read', ret:0x11000, size:0x10, desched:1 }
      { syscall:'read', ret:0x59a, size:0x5aa, desched:1 }
    }
    {
      real_time:8147.770183 global_time:33968, event:`SYSCALL: brk' (state:ENTERING_SYSCALL) tid:32354, ticks:319767963
    rax:0xffffffffffffffda rbx:0x5617be8bb000 rcx:0xffffffffffffffff rdx:0x0 rsi:0x7f967e9f6ac0 rdi:0x5617be8bb000 rbp:0x5617be89a000 rsp:0x7ffe0e340808 r8:0x20 r9:0x1 r10:0x5617be899fe0 r11:0x246 r12:0x7f967e9fe090 r13:0x1000 r14:0x40 r15:0x21000 rip:0x7f967e92305b eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xc fs_base:0x7f967efd1e80 gs_base:0x0
    }
    {
      real_time:8147.770189 global_time:33969, event:`SYSCALLBUF_RESET' tid:32354, ticks:319767963
    }
    === Start rr backtrace:
    rr(_ZN2rr13dump_rr_stackEv+0x5e)[0x561de6223cce]
    rr(_ZN2rr9GdbServer15emergency_debugEPNS_4TaskE+0xc9)[0x561de60b5609]
    rr(+0xf2850)[0x561de60c9850]
    rr(_ZN2rr21EmergencyDebugOstreamD1Ev+0x8f)[0x561de60c9c0f]
    rr(_ZN2rr13ReplaySession13enter_syscallEPNS_10ReplayTaskERKNS0_15StepConstraintsE+0x152)[0x561de618a1f2]
    rr(_ZN2rr13ReplaySession18try_one_trace_stepEPNS_10ReplayTaskERKNS0_15StepConstraintsE+0xfe)[0x561de61939de]
    rr(_ZN2rr13ReplaySession11replay_stepERKNS0_15StepConstraintsE+0x1c3)[0x561de6193f33]
    rr(_ZN2rr14ReplayTimeline19replay_step_forwardENS_10RunCommandE+0xc1)[0x561de61b27b1]
    rr(_ZN2rr9GdbServer14debug_one_stepERNS_10GdbRequestE+0xb71)[0x561de60b45f1]
    rr(_ZN2rr9GdbServer12serve_replayERKNS0_15ConnectionFlagsE+0x7cb)[0x561de60b511b]
    rr(+0x1ada36)[0x561de6184a36]
    rr(_ZN2rr13ReplayCommand3runERSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS7_EE+0x51c)[0x561de618558c]
    rr(main+0x1dc)[0x561de603bd7c]
    /usr/lib/libc.so.6(+0x27cd0)[0x7fddfbc45cd0]
    /usr/lib/libc.so.6(__libc_start_main+0x8a)[0x7fddfbc45d8a]
    rr(_start+0x25)[0x561de603bfe5]
    === End rr backtrace
    Launch gdb with
      gdb '-l' '10000' '-ex' 'set sysroot /' '-ex' 'target extended-remote 127.0.0.1:32428' /mnt/nobackup/rr/python-0/mmap_clone_5_python3.11
    
  13. added 12 commits that reference this issue on Oct 14, 2026
    d67142b
    574a56f
    471c09a
    1265b80
    b39bd14
    dc07dc5
    46fd1ab
    d2f6b15
    f497a64
    13f4c1c
    76fe392
    1af13d0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions