Add hardware-in-the-loop and fault injection tests

[usmanmehmood55]

Summary

This PR adds a hardware-in-the-loop (HIL) test suite for universal_ble and uses it to harden the Windows implementation.

The fixture is a physical nRF52 DK running purpose-built Zephyr firmware. The firmware lives in the separate universal_ble_hil_firmware repository.

The Windows suite currently has 57 working tests:

• 18 baseline tests for ordinary BLE behavior
• 39 fault injection tests (FIT) for errors, awkward timing, disconnects, stale operations, and recovery

The host controls the fixture over BLE. USB serial is used for firmware logs.

What the tests cover

The baseline suite covers:

• filtered scanning and advertisement data
• service, characteristic, and descriptor discovery
• exact characteristic and descriptor reads and writes
• writes with and without response
• notifications and indications
• subscription, unsubscribe, and resubscribe behavior
• notification ordering and binary integrity
• MTU reporting
• peripheral and host disconnects
• reconnects and repeated connection cycles
• concurrent operations
• connected-device enumeration
• ordinary GATT service changes

The FIT suite arms the firmware with a fault plan before starting an ordinary BLE operation. The fixture can then:

• return ATT read, write, or descriptor errors
• return empty values and values around ATT payload boundaries
• delay an operation beyond the host timeout
• disconnect while an operation is pending
• complete an old operation after the host has reconnected
• race host and peripheral disconnects
• force rapid disconnect and reconnect cycles
• change the GATT service database during reads, writes, and subscription changes
• send notifications with gaps, duplicates, reordered sequence numbers, and mixed payload sizes
• disconnect during notification bursts
• keep notifications queued while the host tears down the connection
• emit a notification while the CCC enable operation is still completing

Most tests also verify recovery. After the injected failure, the test performs another read, write, reconnect, or subscription. This catches cases where the original call fails correctly but leaves the connection or operation queue in a broken state.

These tests go through the complete path: Dart API, operation queue, Pigeon, Windows C++, WinRT, Windows Bluetooth stack, the radio link, and finally the Zephyr GATT server. This gives us coverage which mocks and software-only tests cannot provide.

Windows implementation improvements

The HIL work also led to a larger review of the Windows implementation. Production changes are kept separate from the test fixture and include:

1. In-flight GATT operations now retain the native state they use. Disconnect cleanup and service refresh cannot close objects which are still used by a read, write, descriptor operation, discovery call, or MTU query.
2. Concurrent and stale connection attempts are tracked by generation. An older connection or callback cannot replace or tear down a newer connection for the same device.
3. Duplicate connection requests are collapsed while a connection attempt is already pending.
4. GattServicesChanged is handled. Windows rebuilds the GATT map safely, rejects stale refresh results, waits for active operations, and restores existing subscriptions.
5. Notification handlers are registered before enabling the CCC descriptor. A peripheral can therefore send its first notification immediately without that value being lost.
6. Notification changes are transactional. Failed CCC writes roll back provisional handlers, overlapping subscription changes are rejected, and handlers belonging to obsolete connections are ignored.
7. Descriptor reads and writes now retain operation state, report GATT errors correctly, detect disconnects, and use descriptor-specific errors. The Dart readDescriptor API also returns the descriptor value instead of void.
8. Native callbacks are tracked during shutdown. New callbacks are rejected once teardown starts, while callbacks already running are allowed to finish before plugin state is released. This also covers peripheral-mode callbacks.
9. Scan and device-watcher cleanup now handles partial startup, Bluetooth state changes, stale entries, and WinRT exceptions more consistently.
10. UI-thread wake-ups are coalesced, and queued work is rejected after shutdown.
11. The logger’s active level is now atomic, removing a data race between Flutter calls and native callbacks.
12. Notification byte arrays are normalized at the Dart boundary. Calling value.buffer can no longer expose unrelated bytes from the surrounding Pigeon message.

The deterministic native lifetime logic also has software-only tests, which now run in the Windows CI job.

Bugs found

The suite had caught several real issues:

1. Notification and disconnect stress exposed the Windows native crash and connection-lifecycle problems fixed in Windows: Harden BLE connection lifetime, async callbacks, and notification subscription handling #278. The fixture was also used to verify that fix against real hardware.
2. Windows could lose the first notification because it registered ValueChanged after the CCC enable operation had completed.
3. Notification values could remain views into a larger Pigeon message buffer. The visible bytes were correct, but value.buffer could include unrelated envelope data.
4. Windows did not safely refresh its GATT map after a Service Changed event or preserve subscriptions across that refresh.
5. Descriptor operations were not protected against disconnect and refresh races, and some failures were reported with incorrect error types.

The stale connection, stale completion, descriptor, service refresh, and notification timing tests now cover these paths against the physical fixture.

Running the suite

These are manual hardware tests. They are not intended to run in normal pull-request CI because they require an nRF52 DK and exclusive access to a Bluetooth adapter.

Generated Android, Web, and Windows runner files are not committed. They can be recreated with flutter create, which keeps the PR focused on the actual fixture, tests, and production changes.

The Windows suite should be run when changing:

• connection ownership or cleanup
• GATT operation completion
• descriptor handling
• service refresh behavior
• subscriptions and native event handlers
• callback lifetime
• operation queueing
• Windows BLE lifecycle code

The Web runner provides an interactive path for the portable baseline behavior. The suite also contains skipped placeholders for faults which still need firmware support and proper assertions. Those can be implemented gradually as the fixture develops.

[fotiDim]

@usmanmehmood55 thanks for the PR. Would https://www.adafruit.com/product/3406 or https://www.ebay.de/itm/406283688259 work?

[usmanmehmood55]

@fotiDim technically yes, but both of these boards have different ways of installing/flashing the firmware onto the boards, compared to the standard nRF development kit.

They use UART over the USB port, and need the firmware to be partitioned in their specific bootloader's styles, so I'd have to get one myself and test out the flashing process and do some trial and error on the partition sizes etc.

They also might require a physical button to be pressed during every installation/flashing, which would make CI automation difficult.

Standard nRF52 devkits have onboard flasher/debugger, so it'll work right out of the box, and won't require physical button presses.

I will look into the Adafruit board, it might be easier than the ebay link one. But again, the standard devkit would be the easiest.