Runtime statistics collection panics

After upgrading to the newest `openraft` main, I consistently get panics in this form:

```
    thread '<unnamed>' (80836707) panicked at .../.cargo/git/checkouts/openraft-c9fbe90779948994/aae2eea/openraft/src/base/range_map.rs:68:13:
    RangeMap::record: right_boundary 1 must be > previous 1
    stack backtrace:
[...]
      18:        0x1042b3cec - core::panicking::panic_fmt::ha09fb24313daf262
                                   at /rustc/e408947bfd200af42db322daf0fadfe7e26d3bd1/library/core/src/panicking.rs:80:14
      19:        0x10277e980 - <openraft[258d925ee6271c57]::base::range_map::RangeMap<u64, raft_types[4d542e7a11607d4e]::config::Instant>>::record
                                   at .../.cargo/git/checkouts/openraft-c9fbe90779948994/aae2eea/openraft/src/base/range_map.rs:68:13
      20:        0x102c6b54c - <openraft[258d925ee6271c57]::core::runtime_stats::log_stage::lifecycle_latency::LogStages<raft_types[4d542e7a11607d4e]::config::Instant>>::record_stage
                                   at .../.cargo/git/checkouts/openraft-c9fbe90779948994/aae2eea/openraft/src/core/runtime_stats/log_stage/lifecycle_latency.rs:117:66
      21:        0x102bc8230 - <openraft[258d925ee6271c57]::core::runtime_stats::runtime_stats::RuntimeStats<raft_types[4d542e7a11607d4e]::config::CSConfig<...>::record_log_stage
                                   at .../.cargo/git/checkouts/openraft-c9fbe90779948994/aae2eea/openraft/src/core/runtime_stats/runtime_stats.rs:179:24
      22:        0x102bc8300 - <openraft[258d925ee6271c57]::core::runtime_stats::runtime_stats::RuntimeStats<raft_types[4d542e7a11607d4e]::config::CSConfig<...>::record_log_stage_now
                                   at .../.cargo/git/checkouts/openraft-c9fbe90779948994/aae2eea/openraft/src/core/runtime_stats/runtime_stats.rs:168:14
      23:        0x1025953e4 - <openraft[258d925ee6271c57]::core::raft_core::RaftCore<raft_types[4d542e7a11607d4e]::config::CSConfig<...>, raft_network[4419e3111c63c2ef]::network_factory::NetworkFactory<...>, raft_storage[e1b9b65ec7d09c69]::storage::LogStorage<...>, raft_storage[e1b9b65ec7d09c69]::storage::StateMachine<...>> as openraft[258d925ee6271c57]::runtime::RaftRuntime<raft_types[4d542e7a11607d4e]::config::CSConfig<...>, raft_storage[e1b9b65ec7d09c69]::storage::StateMachine<...>::run_command::{closure#0}
                                   at .../.cargo/git/checkouts/openraft-c9fbe90779948994/aae2eea/openraft/src/core/raft_core.rs:2126:36
      24:        0x102583310 - <openraft[258d925ee6271c57]::core::raft_core::RaftCore<raft_types[4d542e7a11607d4e]::config::CSConfig<...>, raft_network[4419e3111c63c2ef]::network_factory::NetworkFactory<...>, raft_storage[e1b9b65ec7d09c69]::storage::LogStorage<...>, raft_storage[e1b9b65ec7d09c69]::storage::StateMachine<...>::run_engine_commands::{closure#0}::{closure#0}
                                   at .../.cargo/git/checkouts/openraft-c9fbe90779948994/aae2eea/openraft/src/core/raft_core.rs:1116:45
      25:        0x10258d4d4 - <openraft[258d925ee6271c57]::core::raft_core::RaftCore<raft_types[4d542e7a11607d4e]::config::CSConfig<...>, raft_network[4419e3111c63c2ef]::network_factory::NetworkFactory<...>, raft_storage[e1b9b65ec7d09c69]::storage::LogStorage<...>, raft_storage[e1b9b65ec7d09c69]::storage::StateMachine<...>::run_engine_commands::{closure#0}
                                   at .../.cargo/git/checkouts/openraft-c9fbe90779948994/aae2eea/openraft/src/core/raft_core.rs:1096:5
      26:        0x10258b344 - <openraft[258d925ee6271c57]::core::raft_core::RaftCore<raft_types[4d542e7a11607d4e]::config::CSConfig<...>, raft_network[4419e3111c63c2ef]::network_factory::NetworkFactory<...>, raft_storage[e1b9b65ec7d09c69]::storage::LogStorage<...>, raft_storage[e1b9b65ec7d09c69]::storage::StateMachine<...>::process_raft_msg::{closure#0}
                                   at .../.cargo/git/checkouts/openraft-c9fbe90779948994/aae2eea/openraft/src/core/raft_core.rs:1298:44
      27:        0x10257d2b8 - <openraft[258d925ee6271c57]::core::raft_core::RaftCore<raft_types[4d542e7a11607d4e]::config::CSConfig<...>, raft_network[4419e3111c63c2ef]::network_factory::NetworkFactory<...>, raft_storage[e1b9b65ec7d09c69]::storage::LogStorage<...>, raft_storage[e1b9b65ec7d09c69]::storage::StateMachine<...>::runtime_loop::{closure#0}::{closure#0}
                                   at .../.cargo/git/checkouts/openraft-c9fbe90779948994/aae2eea/openraft/src/core/raft_core.rs:1245:81
      28:        0x10258a43c - <openraft[258d925ee6271c57]::core::raft_core::RaftCore<raft_types[4d542e7a11607d4e]::config::CSConfig<...>, raft_network[4419e3111c63c2ef]::network_factory::NetworkFactory<...>, raft_storage[e1b9b65ec7d09c69]::storage::LogStorage<...>, raft_storage[e1b9b65ec7d09c69]::storage::StateMachine<...>::runtime_loop::{closure#0}
                                   at .../.cargo/git/checkouts/openraft-c9fbe90779948994/aae2eea/openraft/src/core/raft_core.rs:1197:5
      29:        0x102589a9c - <openraft[258d925ee6271c57]::core::raft_core::RaftCore<raft_types[4d542e7a11607d4e]::config::CSConfig<...>, raft_network[4419e3111c63c2ef]::network_factory::NetworkFactory<...>, raft_storage[e1b9b65ec7d09c69]::storage::LogStorage<...>, raft_storage[e1b9b65ec7d09c69]::storage::StateMachine<...>::do_main::{closure#0}::{closure#0}
                                   at .../.cargo/git/checkouts/openraft-c9fbe90779948994/aae2eea/openraft/src/core/raft_core.rs:308:40
      30:        0x10259355c - <openraft[258d925ee6271c57]::core::raft_core::RaftCore<raft_types[4d542e7a11607d4e]::config::CSConfig<...>, raft_network[4419e3111c63c2ef]::network_factory::NetworkFactory<...>, raft_storage[e1b9b65ec7d09c69]::storage::LogStorage<...>, raft_storage[e1b9b65ec7d09c69]::storage::StateMachine<...>::do_main::{closure#0}
                                   at .../.cargo/git/checkouts/openraft-c9fbe90779948994/aae2eea/openraft/src/core/raft_core.rs:296:5
      31:        0x1025d5060 - <tracing[d14a6d8d7963c731]::instrument::Instrumented<<openraft[258d925ee6271c57]::core::raft_core::RaftCore<raft_types[4d542e7a11607d4e]::config::CSConfig<...>, raft_network[4419e3111c63c2ef]::network_factory::NetworkFactory<...>, raft_storage[e1b9b65ec7d09c69]::storage::LogStorage<...>, raft_storage[e1b9b65ec7d09c69]::storage::StateMachine<...>::do_main::{closure#0}> as core[8abbb8f79ad13dba]::future::future::Future>::poll
                                   at .../.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tracing-0.1.44/src/instrument.rs:321:15
      32:        0x102591d68 - <openraft[258d925ee6271c57]::core::raft_core::RaftCore<raft_types[4d542e7a11607d4e]::config::CSConfig<...>, raft_network[4419e3111c63c2ef]::network_factory::NetworkFactory<...>, raft_storage[e1b9b65ec7d09c69]::storage::LogStorage<...>, raft_storage[e1b9b65ec7d09c69]::storage::StateMachine<...>::main::{closure#0}
                                   at .../.cargo/git/checkouts/openraft-c9fbe90779948994/aae2eea/openraft/src/core/raft_core.rs:268:62
      33:        0x1025d4f88 - <tracing[d14a6d8d7963c731]::instrument::Instrumented<<openraft[258d925ee6271c57]::core::raft_core::RaftCore<raft_types[4d542e7a11607d4e]::config::CSConfig<...>, raft_network[4419e3111c63c2ef]::network_factory::NetworkFactory<...>, raft_storage[e1b9b65ec7d09c69]::storage::LogStorage<...>, raft_storage[e1b9b65ec7d09c69]::storage::StateMachine<...>::main::{closure#0}> as core[8abbb8f79ad13dba]::future::future::Future>::poll
                                   at .../.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tracing-0.1.44/src/instrument.rs:321:15
```

**To Reproduce**

Run a workload with the newest `openraft`, I guess?

Our workload causing this is quite simple - it's basically a persistent K/V store with CRUD in an in-memory simulated cluster environment with 3 nodes (i.e., **the latencies are very low**).

Commenting out the `runtime-stats` feature mitigates the problem (at the cost of not collecting statistics).

**Env (please complete the following information):**
- Openraft version `2c4969e8f5c69a0288f44839a448b9d8e4eb9b57`
- Is the bug still there in the latest version? It seems to be only in `main`.
- Rust-toolchain: 1.94.1
- OS: MacOS 26.4 as well as Linux SLES 15 SP6
- CPU: m3 and x86_64


Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Runtime statistics collection panics #1710

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Runtime statistics collection panics #1710

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions