Skip to content

udev: prevent interruption by thread cancellation - #24

Open
wuguanghao3 wants to merge 1 commit into
openSUSE:queuefrom
wuguanghao3:master
Open

udev: prevent interruption by thread cancellation#24
wuguanghao3 wants to merge 1 commit into
openSUSE:queuefrom
wuguanghao3:master

Conversation

@wuguanghao3

Copy link
Copy Markdown

Using mutexes only guarantees no concurrency during normal runtime. However, if a thread is cancelled mid-execution, a libudev API call might be interrupted halfway (partially executed), leaving memory values in an inconsistent or abnormal state.

Furthermore, this change aligns with the requirement that libudev interfaces are not thread-safe and must not be called from multiple threads.

Using mutexes only guarantees no concurrency during normal runtime.
However, if a thread is cancelled mid-execution, a libudev API call
might be interrupted halfway (partially executed), leaving memory
values in an inconsistent or abnormal state.

Furthermore, this change aligns with the requirement that libudev
interfaces are not thread-safe and must not be called from multiple
threads.

Signed-off-by: Wu Guanghao <wuguanghao3@huawei.com>
@mwilck

mwilck commented Jul 20, 2026

Copy link
Copy Markdown
Collaborator

Thanks! I have applied your patch to my tip branch, and applied another patch on top to wrap all libudev calls with macros, as proposed in opensvc#153.

@mwilck

mwilck commented Jul 20, 2026

Copy link
Copy Markdown
Collaborator

@wuguanghao3, @bmarzins, may I ask you to review 58519b1 ?

@wuguanghao3

wuguanghao3 commented Jul 20, 2026

Copy link
Copy Markdown
Author

@wuguanghao3, @bmarzins, may I ask you to review 58519b1 ?

LGTM

But I have a question: can't we just use (call) directly instead of udev_xxx(arg1, arg2, ...)? Do we need to use different macros based on the parameters? What is the advantage of doing it this way?

#define MT_WRAP(call) { \
   ...\
   __ret = (call);  
   ...\
}

MT_WRAP((udev_xxx)(a))
MT_WRAP((udev_yyy)(a,b))
...

@mwilck

mwilck commented Jul 21, 2026

Copy link
Copy Markdown
Collaborator

But I have a question: can't we just use (call) directly instead of udev_xxx(arg1, arg2, ...)?

In principle we could. We wouldn't need the mt_udev... wrappers at all, we'd just wrap every call in-place. I recall that we had some weird problems with nested pthread_cleanup_push() calls with some old compilers in the past.

Let me try.

@mwilck

mwilck commented Jul 21, 2026

Copy link
Copy Markdown
Collaborator

I remember the rationale now. @wuguanghao3, your suggestion works only with preprocessor macros. For wrapper functions it's either impossible or really clumsy, because we need to insert the argument types in the wrapper function's argument list, while omitting them in the actual libudev call.

(If you can conceive of an elegant way to do this, let me know).

What we can do is just use preprocessor macros:

#define MT_UDEV_WRAP(call)						\
	({								\
		int __oldstate;						\
		pthread_setcancelstate(PTHREAD_CANCEL_DISABLE, &__oldstate); \
		pthread_mutex_lock(&libudev_mutex);			\
		__typeof__(call) __ret = (call);			\
		pthread_mutex_unlock(&libudev_mutex);			\
		pthread_setcancelstate(__oldstate, NULL);		\
		pthread_testcancel();					\
		__ret;							\
	})

#define udev_ref(udev) MT_UDEV_WRAP(udev_ref(udev))
#define udev_list_entry_get_by_name(list_entry, name) \
        MT_UDEV_WRAP(udev_list_entry_get_by_name(list_entry, name))
...

This works, and we don't even need any mt_udev_ wrapper symbols. We just need to make libudev_mutex globally visible.

However, we have the wrapper functions for a reason. By introducing the wrapper functions, we achieve clean separation from libudev, and we can make sure that we don't ever call libudev directly. We just don't include libudev.h anywhere except in mt-libudev.c1. When we use the preprocessor-only approach, we have much less separation; we need to include libudev.h basically everywhere, and make libudev_mutex globally visible.
Moreover, I don't like inlining the cancellation prevention code in every libudev call; IMO it's an example of "bad" inlining. But that's mainly a matter of taste.

Footnotes

  1. This makes me realize that we've actually got this wrong in some recent commits!

@mwilck

mwilck commented Jul 21, 2026

Copy link
Copy Markdown
Collaborator

I realize what I did previously here was wrong. By applying my commit on top of yours, I called pthread_setcancelstate() twice (actually 4 times!) now for every udev call.

I've pushed a new version to my tip branch (b2d35f7). @wuguanghao3, please review test it in your environment.

And again, if you can conceive of a simpler way to write these wrappers, let me know.

@mwilck

mwilck commented Jul 21, 2026

Copy link
Copy Markdown
Collaborator

I had to rebase and re-push again, current commit ID is 3019b25

@bmarzins bmarzins left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me.

@wuguanghao3

Copy link
Copy Markdown
Author

I had to rebase and re-push again, current commit ID is 3019b25

OK, LGTM

@mwilck

mwilck commented Jul 22, 2026

Copy link
Copy Markdown
Collaborator

@wuguanghao3

OK, LGTM

Have you run your test with this code?

@wuguanghao3

Copy link
Copy Markdown
Author

@wuguanghao3

OK, LGTM

Have you run your test with this code?

Yes, currently testing. The issue has not recurred yet.

@wuguanghao3

Copy link
Copy Markdown
Author

@wuguanghao3

OK, LGTM

Have you run your test with this code?

Yes, currently testing. The issue has not recurred yet.

The UAF issue could not be reproduced. However, a new reference issue occurred, which appears to be exclusive to version 0.14.3 and was not present in 0.9.5. I am currently running tests on 0.14.3 with the current patch applied.

(gdb) bt
#0  0x00007f27a3b97bac in ?? () from /usr/lib64/libc.so.6
#1  0x00007f27a3b4bb86 in raise () from /usr/lib64/libc.so.6
#2  0x00007f27a3b363fc in abort () from /usr/lib64/libc.so.6
#3  0x00007f27a3dfd8ca in log_assert_failed (text=<optimized out>, file=<optimized out>, line=<optimized out>, func=<optimized out>) at ../src/basic/log.c:976
#4  0x00007f27a3e00507 in udev_device_unref (p=<optimized out>) at ../src/libudev/libudev-device.c:508
#5  udev_device_unref (p=p@entry=0x556220ff4d70) at ../src/libudev/libudev-device.c:508
#6  0x00007f27a3e362af in mt_udev_device_unref (__arg1=0x556220ff4d70) at mt-libudev.c:80
#7  0x00007f27a3ebff5d in free_path (pp=0x556220ffa060) at structs.c:179
#8  0x00007f27a3ebfffd in free_pathvec (vec=0x556220db2d90, free_paths=<optimized out>) at structs.c:201
#9  0x000055620cde21b8 in cleanup_paths (vecs=0x556220d90620) at main.c:3792
#10 cleanup_vecs () at main.c:3807
#11 cleanup_child () at main.c:3916
#12 0x00007f27a3b4e135 in ?? () from /usr/lib64/libc.so.6
#13 0x00007f27a3b4e230 in exit () from /usr/lib64/libc.so.6
#14 0x00007f27a3b37617 in ?? () from /usr/lib64/libc.so.6
#15 0x00007f27a3b376c9 in __libc_start_main () from /usr/lib64/libc.so.6
#16 0x000055620cde15d5 in _start ()

@mwilck

mwilck commented Jul 23, 2026

Copy link
Copy Markdown
Collaborator

Sigh. This indicates a refcount underflow on the udev_device. Which is basically the same thing as a UAF.

Could you perhaps apply the current commit to 0.9.5 and see if it's really a regression between 0.95 and 0.14.3? Given that the issue has apparently occured after 4h of testing, you proabably need to run the test for a couple of hours to verify that the problem does not occur.

@wuguanghao3

Copy link
Copy Markdown
Author

Sigh. This indicates a refcount underflow on the udev_device. Which is basically the same thing as a UAF.

Could you perhaps apply the current commit to 0.9.5 and see if it's really a regression between 0.95 and 0.14.3? Given that the issue has apparently occured after 4h of testing, you proabably need to run the test for a couple of hours to verify that the problem does not occur.

The UAF and the reference issue are distinct. The UAF involves memory inconsistency resulting from thread cancellation, whereas the reference issue may stem from a redundant execution of an upper-level call.
During a 24-hour test of the initial patch on version 0.9.5, the issue could not be reproduced. Given that the underlying logic remains identical between the final and initial versions, there is a high probability that this issue was introduced in a release between 0.9.5 and 0.14.3. I will conduct further testing on version 0.9.5 utilizing the latest proposed solution.

@wuguanghao3

Copy link
Copy Markdown
Author

Sigh. This indicates a refcount underflow on the udev_device. Which is basically the same thing as a UAF.
Could you perhaps apply the current commit to 0.9.5 and see if it's really a regression between 0.95 and 0.14.3? Given that the issue has apparently occured after 4h of testing, you proabably need to run the test for a couple of hours to verify that the problem does not occur.

The UAF and the reference issue are distinct. The UAF involves memory inconsistency resulting from thread cancellation, whereas the reference issue may stem from a redundant execution of an upper-level call. During a 24-hour test of the initial patch on version 0.9.5, the issue could not be reproduced. Given that the underlying logic remains identical between the final and initial versions, there is a high probability that this issue was introduced in a release between 0.9.5 and 0.14.3. I will conduct further testing on version 0.9.5 utilizing the latest proposed solution.

It has been confirmed that this issue was introduced in a release between 0.9.5 and 0.14.3. I conducted a 7-hour test running version 0.9.5 with the latest commit, and the issue could not be reproduced.

@mwilck

mwilck commented Jul 24, 2026

Copy link
Copy Markdown
Collaborator

The UAF involves memory inconsistency resulting from thread cancellation, whereas the reference issue may stem from a redundant execution of an upper-level call.

A refcount underflow means that an object is being unref'd after having reached refcount zero. IOW, this object was still in use after it should have been freed already. Maybe the free() hadn't happened yet, or for some reason libasan didn't catch the problem. IMO it is pure "luck" that the memory where the refcount is stored is still accessible and contains a negative value.

How often have you observed this refcount underflow so far?
Do you have log files (ideally with verbosity 3) where we can see what happened before the crash?

It has been confirmed that this issue was introduced in a release between 0.9.5 and 0.14.3

There has been a major cleanup of the code that frees maps and paths in 0.14.0 (commit edf2bac ff.) So if you want to to investigate further, you may want to start with comparing 0.13.0 and 0.14.0, both with the latest wrapper patch applied. When you apply the patch, make sure that no direct libudev calls remain.

Given that you observe the problem in the exit() code flow, while multipathd frees its internal data structures, the easiest fix would be to just skip the exit handlers, the only purpose of which is to show zero memory leaks when multipathd is run under a leak checking program. This would have no negative consequences for running multipathd in production, because the kernel frees the entire memory used by the process anyway. But it's not a solution we can generally apply, because it would strongly affect our ability to assess multipathd for memory leaks. In theory, we could just skip the call to cleanup_vecs() in the atexit handler in "production" builds. But obviously I'd prefer to find the actual problem.

Can you share your test program, so that others can try to reproduce the issue and find a solution? I have also done some extensive testing, but apparently the way you restart multipathd during the test can trigger error conditions that I was not seeing.

@wuguanghao3

Copy link
Copy Markdown
Author

The UAF involves memory inconsistency resulting from thread cancellation, whereas the reference issue may stem from a redundant execution of an upper-level call.

A refcount underflow means that an object is being unref'd after having reached refcount zero. IOW, this object was still in use after it should have been freed already. Maybe the free() hadn't happened yet, or for some reason libasan didn't catch the problem. IMO it is pure "luck" that the memory where the refcount is stored is still accessible and contains a negative value.

How often have you observed this refcount underflow so far? Do you have log files (ideally with verbosity 3) where we can see what happened before the crash?

It has been confirmed that this issue was introduced in a release between 0.9.5 and 0.14.3

There has been a major cleanup of the code that frees maps and paths in 0.14.0 (commit edf2bac ff.) So if you want to to investigate further, you may want to start with comparing 0.13.0 and 0.14.0, both with the latest wrapper patch applied. When you apply the patch, make sure that no direct libudev calls remain.

Given that you observe the problem in the exit() code flow, while multipathd frees its internal data structures, the easiest fix would be to just skip the exit handlers, the only purpose of which is to show zero memory leaks when multipathd is run under a leak checking program. This would have no negative consequences for running multipathd in production, because the kernel frees the entire memory used by the process anyway. But it's not a solution we can generally apply, because it would strongly affect our ability to assess multipathd for memory leaks. In theory, we could just skip the call to cleanup_vecs() in the atexit handler in "production" builds. But obviously I'd prefer to find the actual problem.

Can you share your test program, so that others can try to reproduce the issue and find a solution? I have also done some extensive testing, but apparently the way you restart multipathd during the test can trigger error conditions that I was not seeing.

Below is a simplified test case that includes the main operations. One additional point to note: we identify local disks as multipath by default.

CONFIG_CONTENT='devices {
    device {
        vendor "(LIO-ORG)"
        product ".*"
        no_path_retry "fail"
    }
}'

function prepare_local_target() {
    local count=100
    local testdir=$1
    mkdir -p "$testdir"
    i=1
    while [ $i -le $count ]; do
        dd if=/dev/zero of="$testdir"/disk$i count=10 bs=1M
        ((i = i + 1))
    done

    i=1
    echo -e "clearconfig confirm=true\n" | targetcli
    echo -e "cd /backstores/fileio\n" | targetcli
    while [ $i -le $count ]; do
        echo -e "create file_or_dev=$testdir/disk$i name=filedisk$i size=100M\n" | targetcli
        ((i = i + 1))
    done
    echo -e "cd /iscsi\ncreate iqn.2003-01.org.linux-iscsi.localhost.test:sn.ff69cf92aac7\n" | targetcli
    echo -e "cd iqn.2003-01.org.linux-iscsi.localhost.test:sn.ff69cf92aac7/tpg1/luns\n" | targetcli
    i=1
    while [ $i -le $count ]; do
        echo -e "create /backstores/fileio/filedisk$i lun=$i\n" | targetcli
        ((i = i + 1))
    done
    echo -e "cd /iscsi/iqn.2003-01.org.linux-iscsi.localhost.test:sn.ff69cf92aac7/tpg1/acls\n" | targetcli
    echo -e "create iqn.2003-01.org.linux-iscsi.localhost.test:sn.ff69cf92aac7:client\n" | targetcli
    systemctl restart target

    echo "InitiatorName=iqn.2003-01.org.linux-iscsi.localhost.test:sn.ff69cf92aac7:client" > /etc/iscsi/initiatorname.iscsi
    systemctl restart iscsid
}

function udev_trigger() {
    while true; do
        for i in $(seq 1 20); do
            udevadm trigger
        done
        sleep 1
    done
}

function pre_test() {
    yum install -y targetcli open-iscsi multipath-tools target-restore scsi-target-utils || exit "${ETS_FAILED}"
    cp /etc/iscsi/initiatorname.iscsi .
    prepare_local_target /home/fileio-$$

    iscsiadm -m node --logout ALL
    iscsiadm -m node --op delete ALL
    iscsiadm -m discovery -p 127.0.0.1 -t st || exit "${ETS_FAILED}"
    iscsiadm -m node -p 127.0.0.1 -l || exit "${ETS_FAILED}"
    echo remove_local_disk=1 > /etc/multipath_private.conf
    version_compare "$(rpm -q --qf '%{version}' multipath-tools)" ">=" "0.9.5" && echo "$CONFIG_CONTENT" > /etc/multipath.conf
}

function do_test() {
    ulimit -c unlimited
    version_compare "$(rpm -q --qf '%{version}' multipath-tools)" ">=" "0.9.5" && modprobe dm-multipath
    systemctl restart multipathd || exit "${ETS_FAILED}"
    udev_trigger &> /dev/null &
    pid_udev=$!

    time=$(date "+%Y-%m-%d %H:%M:%S")
    for loop in $(seq 1 20); do
        for i in $(seq 1 5); do
            systemctl restart multipathd
        done
        kill -9 "$(pidof /sbin/multipathd)"

        coredumpctl list -S "$time" | grep multipathd
        if [ $? -eq 0 ]; then
            coredumpctl info
            kill -9 "$pid_udev"
            add_failure "multipathd segfault"
            return "${ETS_FAILED}"
        fi
    done
    kill -9 "$pid_udev"
}

function post_test() {
    iscsiadm -m node -p 127.0.0.1 -u
    iscsiadm -m node -o delete -p 127.0.0.1
    multipath -F
    systemctl stop multipathd
    systemctl stop iscsid
    systemctl stop target
    cp initiatorname.iscsi /etc/iscsi/initiatorname.iscsi
    rm -rf /home/fileio-$$
}

@wuguanghao3

Copy link
Copy Markdown
Author

The UAF involves memory inconsistency resulting from thread cancellation, whereas the reference issue may stem from a redundant execution of an upper-level call.

A refcount underflow means that an object is being unref'd after having reached refcount zero. IOW, this object was still in use after it should have been freed already. Maybe the free() hadn't happened yet, or for some reason libasan didn't catch the problem. IMO it is pure "luck" that the memory where the refcount is stored is still accessible and contains a negative value.
How often have you observed this refcount underflow so far? Do you have log files (ideally with verbosity 3) where we can see what happened before the crash?

It has been confirmed that this issue was introduced in a release between 0.9.5 and 0.14.3

There has been a major cleanup of the code that frees maps and paths in 0.14.0 (commit edf2bac ff.) So if you want to to investigate further, you may want to start with comparing 0.13.0 and 0.14.0, both with the latest wrapper patch applied. When you apply the patch, make sure that no direct libudev calls remain.
Given that you observe the problem in the exit() code flow, while multipathd frees its internal data structures, the easiest fix would be to just skip the exit handlers, the only purpose of which is to show zero memory leaks when multipathd is run under a leak checking program. This would have no negative consequences for running multipathd in production, because the kernel frees the entire memory used by the process anyway. But it's not a solution we can generally apply, because it would strongly affect our ability to assess multipathd for memory leaks. In theory, we could just skip the call to cleanup_vecs() in the atexit handler in "production" builds. But obviously I'd prefer to find the actual problem.
Can you share your test program, so that others can try to reproduce the issue and find a solution? I have also done some extensive testing, but apparently the way you restart multipathd during the test can trigger error conditions that I was not seeing.

Below is a simplified test case that includes the main operations. One additional point to note: we identify local disks as multipath by default.

CONFIG_CONTENT='devices {
    device {
        vendor "(LIO-ORG)"
        product ".*"
        no_path_retry "fail"
    }
}'

function prepare_local_target() {
    local count=100
    local testdir=$1
    mkdir -p "$testdir"
    i=1
    while [ $i -le $count ]; do
        dd if=/dev/zero of="$testdir"/disk$i count=10 bs=1M
        ((i = i + 1))
    done

    i=1
    echo -e "clearconfig confirm=true\n" | targetcli
    echo -e "cd /backstores/fileio\n" | targetcli
    while [ $i -le $count ]; do
        echo -e "create file_or_dev=$testdir/disk$i name=filedisk$i size=100M\n" | targetcli
        ((i = i + 1))
    done
    echo -e "cd /iscsi\ncreate iqn.2003-01.org.linux-iscsi.localhost.test:sn.ff69cf92aac7\n" | targetcli
    echo -e "cd iqn.2003-01.org.linux-iscsi.localhost.test:sn.ff69cf92aac7/tpg1/luns\n" | targetcli
    i=1
    while [ $i -le $count ]; do
        echo -e "create /backstores/fileio/filedisk$i lun=$i\n" | targetcli
        ((i = i + 1))
    done
    echo -e "cd /iscsi/iqn.2003-01.org.linux-iscsi.localhost.test:sn.ff69cf92aac7/tpg1/acls\n" | targetcli
    echo -e "create iqn.2003-01.org.linux-iscsi.localhost.test:sn.ff69cf92aac7:client\n" | targetcli
    systemctl restart target

    echo "InitiatorName=iqn.2003-01.org.linux-iscsi.localhost.test:sn.ff69cf92aac7:client" > /etc/iscsi/initiatorname.iscsi
    systemctl restart iscsid
}

function udev_trigger() {
    while true; do
        for i in $(seq 1 20); do
            udevadm trigger
        done
        sleep 1
    done
}

function pre_test() {
    yum install -y targetcli open-iscsi multipath-tools target-restore scsi-target-utils || exit "${ETS_FAILED}"
    cp /etc/iscsi/initiatorname.iscsi .
    prepare_local_target /home/fileio-$$

    iscsiadm -m node --logout ALL
    iscsiadm -m node --op delete ALL
    iscsiadm -m discovery -p 127.0.0.1 -t st || exit "${ETS_FAILED}"
    iscsiadm -m node -p 127.0.0.1 -l || exit "${ETS_FAILED}"
    echo remove_local_disk=1 > /etc/multipath_private.conf
    version_compare "$(rpm -q --qf '%{version}' multipath-tools)" ">=" "0.9.5" && echo "$CONFIG_CONTENT" > /etc/multipath.conf
}

function do_test() {
    ulimit -c unlimited
    version_compare "$(rpm -q --qf '%{version}' multipath-tools)" ">=" "0.9.5" && modprobe dm-multipath
    systemctl restart multipathd || exit "${ETS_FAILED}"
    udev_trigger &> /dev/null &
    pid_udev=$!

    time=$(date "+%Y-%m-%d %H:%M:%S")
    for loop in $(seq 1 20); do
        for i in $(seq 1 5); do
            systemctl restart multipathd
        done
        kill -9 "$(pidof /sbin/multipathd)"

        coredumpctl list -S "$time" | grep multipathd
        if [ $? -eq 0 ]; then
            coredumpctl info
            kill -9 "$pid_udev"
            add_failure "multipathd segfault"
            return "${ETS_FAILED}"
        fi
    done
    kill -9 "$pid_udev"
}

function post_test() {
    iscsiadm -m node -p 127.0.0.1 -u
    iscsiadm -m node -o delete -p 127.0.0.1
    multipath -F
    systemctl stop multipathd
    systemctl stop iscsid
    systemctl stop target
    cp initiatorname.iscsi /etc/iscsi/initiatorname.iscsi
    rm -rf /home/fileio-$$
}

I attempted to capture the call stack using ASan but was unsuccessful. This indicates that the memory address had not been released yet when the double decrement of the reference count occurred.

@mwilck

mwilck commented Jul 27, 2026

Copy link
Copy Markdown
Collaborator

So you're running both udvadm trigger and systemctl restart multipathd in tight loops, at the same point in time.

I suppose you're aware that this is an unrealistic scenario. Of course it's wrong that multipathd gets the refcount wrong and crashes, but I don't see it as a problem with high practical relevance.

I'll try to reproduce the issue nonetheless. Regardless, I'd appreciate a journal output from your test showing the sequence of events before the crash.

Because multipathd is restarted immediately after startup, It's likely that the issue is related to the initial startup and device configuration, and to being interrupted / killed in this stage.

@mwilck

mwilck commented Jul 27, 2026

Copy link
Copy Markdown
Collaborator

Your test doesn't work on my system. systemd simply refuses to restart multipathd in such a quick succession (which is reasonable).

Job for multipathd.service failed because start of the service was attempted too often.
See "systemctl status multipathd.service" and "journalctl -xeu multipathd.service" for details.
To force a start use "systemctl reset-failed multipathd.service"

At least some sleep time is needed between multipathd restarts.

@mwilck

mwilck commented Jul 27, 2026

Copy link
Copy Markdown
Collaborator

I've made your test work by adding systemctl reset-failed multipathd to the restart loop. But I haven't been able to reproduce the core dump. My current test system is pretty different from yours, though, I was running the test on a system with real multipath hardware.

@mwilck
mwilck force-pushed the queue branch 7 times, most recently from 27ed4e7 to 6931e84 Compare July 28, 2026 15:11
@bmarzins

bmarzins commented Aug 4, 2026

Copy link
Copy Markdown

For what it's worth, the commit for this in your tip branch looks good to me.

@wuguanghao3

Copy link
Copy Markdown
Author

I've made your test work by adding systemctl reset-failed multipathd to the restart loop. But I haven't been able to reproduce the core dump. My current test system is pretty different from yours, though, I was running the test on a system with real multipath hardware.

Sorry for the delay, I've been a bit busy recently and haven't had a chance to look into this. I'll try to reproduce it and find the root cause when I have some time, and will keep you updated.

@bmarzins

bmarzins commented Aug 6, 2026

Copy link
Copy Markdown

@wuguanghao3 I copied your setup, and have reproduced the same coredump. Unfortunately, it took hours running overnight to hit it.

@bmarzins

bmarzins commented Aug 6, 2026

Copy link
Copy Markdown

@wuguanghao3 are you able to quickly and reliably hit this ref count assert? Could you try removing the
pthread_testcancel(); lines in the LU_WRAP_* functions from libmpathutil/mt-libudev.c, and seeing if that fixes the problem. I haven't found exactly where this could make pp->udev get double-unref'ed, but I have found a place where a different udev_device object can get double-unref'ed. In path_discovery(), if the thread gets cancelled here

udev_device_unref(udevice);
That pthread_testcancel() will trigger after the udev device is unref'ed and run the cleanup handlers, including cleanup_udev_device_ptr() which will unref the device again. If the thread didn't get cancelled, the next line would set udevice to NULL so the cleanup handler wouldn't do anything, but with that pthread_testcancel(), the next line never gets run.

The general problem with it is that it makes all the udev_* functions become pthread cancellation points (after they have done their work), and the existing code doesn't account for this.

@bmarzins

bmarzins commented Aug 6, 2026

Copy link
Copy Markdown

So, I'm pretty sure I was right about pthread_testcancel() being the problem. Your test triggers a bunch of uevents while repeatedly shutting down multipathd. In both path uevent handling functions, uev_add_path() and uev_update_path(), there is this code like this

udev_device_unref(pp->udev);
pp->udev = udev_device_ref(uev->udev);

If the thread is cancelled (because multipathd is shutting down) while udev_device_unref(pp->udev) is being called, pp->udev will still point to the old, unref'ed value when cleanup_paths() is called.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants