Skip to content

driver/python_can: fix the way recv and flush_tx_buffer are called - #78

Merged
tridge merged 2 commits into
dronecan:masterfrom
oystub:oystub/fix-pythoncan-timeout
Sep 17, 2026
Merged

tridge merged 2 commits into
dronecan:masterfrom
oystub:oystub/fix-pythoncan-timeout

Conversation

@oystub

@oystub oystub commented Dec 18, 2024 •

Copy link
Copy Markdown
Contributor

python-can's recv function is supposed to take a timeout in seconds or None, not in milliseconds or -1.
See issue #77

In addition, in newer versions of python-can, the flush_tx_buffer raises a NotImplementedError for many interface types like socketcan. I raised an issue requesting a change in this behaviour here: hardbyte/python-can#1922

This combination makes pydronecan unresponsive, and not working for send-operations at all if python-can is installed on the system, so I think a workaround like this is warranted.

@oystub oystub changed the title driver/python_can: use correct timeout for pyhon-can recv() driver/python_can: fix the way recv and flush_tx_buffer are called Feb 22, 2025

@fallenmi fallenmi left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Codex-assisted review.

Reviewed exact head 7d128572aba5be886213d76758c62635dbdb7386. Both transport changes match python-can's contract: recv() accepts seconds or None, and the base flush_tx_buffer() reports unsupported backends with NotImplementedError. The patch passes the timeout through unchanged and catches only that unsupported-operation signal, leaving unexpected write errors on the existing feedback path.

A focused fake-bus oracle changed from recv_timeouts=[12.5, -1] and failed TX feedback on the base to recv_timeouts=[0.0125, None] and successful TX feedback on both the exact head and current merge. A separate RuntimeError flush case still propagated on all trees. The full repository suite passed 58/58 on the PR base, exact head, and current merge; diff and compile checks are clean.

python-can's recv function is supposed to take a timeout in seconds, or None, not in milliseconds or -1.
E.g. the socketcan interface does not have `flush_tx_buffer` implemented. Calling the function without a `try` would cause send operations to fail.
@tridge
tridge force-pushed the oystub/fix-pythoncan-timeout branch from 7d12857 to 54a2299 Compare September 17, 2026 10:58
@AP-Review

Copy link
Copy Markdown

Automated review note — AI-generated (Claude), validated against the live diff. Please sanity-check before acting.

Reviewed at head 54a22990ef. This is the first review of this PR by this workflow.

Full report: https://uav.tridgell.net/DevCallReviews/2026_09_17_AIReview/devcall_pr_reviews.html#prpydronecan-78

Verdict: APPROVE — no blockers

Four lines, both premises correct, and each proved by mutation against the real driver rather than by reading. Two independent passes reached APPROVE.

Both premises verified by running the code. Running the real driver at your head and at its base 622229f against python-can's virtual backend:

probe base 622229f head 54a22990ef
node.spin(0.01) 9.997 s 0.010 s
send_frame() then spin() raised NotImplementedError OK
driver.receive() (timeout=None) ValueError, swallowed, returned in 0.8 ms blocks correctly

A second independent pass reproduced all of this across five python-can versions (4.2.2, 4.3.1, 4.4.0, 4.5.0, 4.6.1) and measured 9.996094 s against 0.010126 s.

Premise 1 — the *1000 / -1 conversion was wrong. can.BusABC.recv takes seconds and returns None on timeout. The deleted line was copy-pasted from dronecan/driver/socketcan.py, which has the identical line — but there it feeds select.poll(), which genuinely takes milliseconds and treats -1 as infinite. Since node.py:431 clamps read_timeout to [0, 1] seconds, the old code could block for up to 1000 seconds per spin iteration, which is exactly the unresponsive symptom in the linked issue. The None case was broken too, not merely different: it became recv(-1), and socketcan's select.select(..., -1) raises ValueError, which the blanket except Exception swallowed into a busy-spin. Blocking on timeout=None now matches slcan.py, mcast.py and mavcan.py, and matches node.spin's own documented "If None, the method will never return."

Premise 2 — flush_tx_buffer really does break TX. can/bus.py:465 in 4.6.1 raises NotImplementedError. Bisected: 4.3.1 is a no-op and 4.4.0 onward raises — your PR body says "after 4.4.0", but it is 4.4.0 itself, which may be worth correcting upstream. Backends with no override (socketcan, slcan, serial, usb2can, virtual) therefore hit it, and the exception landed in the except Exception at python_can.py:116, was pushed onto _write_feedback_queue and re-raised in the caller's thread with a misleading traceback.

One correction to be fair to the old code: the virtual peer did still receive the frame before the flush raised, so "all sending was broken" overstates the transmission effect — what actually broke was that every send surfaced a spurious exception to the caller.

The fix is correctly scoped: the try wraps only flush_tx_buffer, so genuine send() failures still reach the feedback queue and the OSError/ENOBUFS retry path at :109 is untouched. The change is backend-neutral or better everywhere.

Notes, none blocking

  • dronecan/driver/python_can.py:103 — calling flush_tx_buffer() after every send() is the wrong call on the backends where it works. Its contract is "discard every message that may be queued in the output buffer" — a cancel, not a drain — so on an implementing backend it risks discarding the frame just queued. kvaser/canlib.py:648 issues canIOCTL_FLUSH_TX_BUFFER, etas is OCI_ResetQueue(txQueue), systec is RESET_ONLY_TX_BUFF, and worst, vector/canlib.py:934's own docstring warns it "will flush the queue and send a high voltage message (ID = 0, DLC = 0, no data)" — so on a Vector adapter pydronecan injects a spurious frame for every frame it transmits. Codex called the real Vector flush with a mocked XL API and confirmed it requests one ID-0/DLC-0 transmission. My suggestion for a follow-up would be to delete the call rather than guard it: git log -L shows it has been there since the driver was added with no rationale, and the guarded version is already a no-op on the backends ArduPilot users actually hit. That is a separate question and should not hold this PR.
  • python_can.py:44 — bustype= is deprecated since python-can 4.2.0 and scheduled for removal in 5.0.0 (confirmed by capturing the runtime warning). Same class of API drift this PR fixes; it is the next thing that will break this driver.
  • python_can.py:87 — can.__version__ >= '4.0.0' is a string comparison, so '10.0.0' >= '4.0.0' is False and the is_extended_id/extended_id selection silently inverts at python-can 10. Pre-existing and very low priority; packaging.version.parse would be correct.
  • There is no test coverage for this driver at all — no reference to PythonCAN, python_can or import can anywhere in test/, and test/test_driver.py:14 is still a TODO. A regression test is feasible using the virtual backend behind a pytest.importorskip("can"), since python-can is not in install_requires. Optional; I would not hold the PR for it.

Dropped as a false positive during validation: I had suggested also catching AttributeError in case python-can ever removes the method. The second pass refuted it — every tested version exposes BusABC.flush_tx_buffer(), no removal is proposed, and catching the documented unsupported-operation exception is the right thing.

Checked clean: API contracts read from the installed library rather than from memory (inspect.signature/getsource at 4.6.1), the full recv retry loop, and socketcan's handling of the old -1. All three callers checked (node.py:431, node.py:442, tools/dronecan_bridge.py:29) — all pass finite timeouts, so the block-forever path is not newly reachable from node.spin. Sibling drivers all take seconds. Nothing new escapes to the caller. Specifically checked for a shutdown hang: with head blocked in receive(None), close() returned in 0.080 s and the writer exited. pytest test gives 58 passed at this head. There is an earlier APPROVED review from fallenmi against head 7d128572; these conclusions were reached independently and agree with it.

@AP-Review

Copy link
Copy Markdown

Automated review note — AI-generated (Claude), validated against the live diff. Please sanity-check before acting.

Re-reviewed on request at the same head 54a22990ef (unchanged since my earlier comment) — no push of yours triggered this, so nothing here is a reaction to new code. Verdict unchanged: APPROVE, no blockers. My earlier comment above still stands and is not superseded; this is an addendum. Both premises were re-measured from scratch rather than carried over, and two fresh Codex agents (one finding-validation, one cold) both returned APPROVE again.

Full re-review: https://uav.tridgell.net/DevCallReviews/followups/2026_09_17_2330/devcall_pr_reviews.html#prpydronecan-78

Five things are new, one of which corrects the earlier comment.

1. CI has never run on this PR — a maintainer has to release it

The single Python package workflow run for this head (run 35213233708, created at 10:58:22Z by the force-push) is status=completed, conclusion=action_required — the first-time-contributor gate, waiting for someone with write access to click Approve and run workflows. Three independent confirmations: gh pr checks says "no checks reported on the branch", commits/<sha>/check-runs returns total_count=0, and commits/<sha>/status is pending with zero statuses. GitHub reports mergeStateStatus=UNSTABLE.

The suite does pass at this head — 58 passed run locally on Python 3.14 with python-can 4.6.1 — but that is my run, not the project's, and it should not substitute for CI before merge.

2. Correction to my earlier comment: close() does not release a thread parked in receive(None)

The earlier comment said "checked for a shutdown hang: with head blocked in receive(None), close() returned in 0.080 s". The close() timing was right but the conclusion drawn from it was wrong — that probe had the blocked reader, a second driver and the peer bus all on one virtual channel, so the frame the second driver sent is what woke the reader, not close().

Re-run on an isolated channel with no traffic at all: reader parked, close() returned in 0.28 ms, reader still parked 2 s later. The cause is in python-can, not in your patch — can/interfaces/virtual.py's VirtualBus.shutdown() removes the queue from the channel registry and never wakes a blocked queue.get(). Both Codex passes reproduced this independently.

Scope, so it is not overstated, and why it does not change the verdict: this is not a regression against working behaviour. At base, receive(None) returned instantly only because recv(-1) raised ValueError and receive()'s blanket except swallowed it — a busy-spin, not a shutdown path. And it matches the sibling drivers: socketcan.py:277-279 parks in poll(-1) and its close() at :269 closes the fd without waking it. No in-tree caller passes None (node.py:431, node.py:442, tools/dronecan_bridge.py:29 are all finite). Worth knowing about; not worth changing here.

3. flush_tx_buffer is not uniformly a "discard" — which sharpens the follow-up I suggested

I framed it earlier as "a cancel, not a drain". Read out of the installed python-can 4.6.1 source, that holds for four backends — kvaser/canlib.py:648 (canIOCTL_FLUSH_TX_BUFFER), etas/__init__.py:251 (OCI_ResetQueue(txQueue)), systec/ucanbus.py:256 (RESET_ONLY_TX_BUFF), vector/canlib.py:934 (discard plus the high-voltage ID 0 / DLC 0 frame its own docstring warns about) — but not for two others:

  • ixxat/canlib_vcinpl.py:659 is canChannelWaitTxEvent(handle, INFINITE) — a blocking wait for the queue to drain, with no timeout. Its own comment reads TODO #64: no timeout?.
  • canalystii.py:197 flushes with float("infinity").

So on IXXAT and CANalyst-II, the per-frame call is an unbounded blocking wait inside the writer thread, not a discard at all. That makes the case for deleting the call stronger than I put it before: the method has no consistent meaning across backends, so calling it after every send() cannot be right for all of them. Still a follow-up, still not a reason to hold this PR.

4. Today's force-push was a pure rebase, so @fallenmi's approval still applies verbatim

The branch was force-pushed at 10:58:20Z (nine seconds after the AIReview label went on) onto current master 622229f. Settled by blob SHA rather than by re-reading diffs: dronecan/driver/python_can.py is blob 2b9ec24db3 at both the previously-approved head 7d128572ab and the current 54a22990ef, and the normalised patch against each head's own merge-base is byte-identical. Nothing in that 2026-08-30 approval needs revisiting.

5. Compatibility confirmed below python-can 4.x too

The earlier pass covered 4.2.2–4.6.1, but the driver still carries a can.__version__ < '4.0.0' branch at python_can.py:87-100. On 3.3.4: BusABC.flush_tx_buffer is a bare pass (bus.py:364) and recv() is documented in seconds (bus.py:64). So the *1000 conversion you deleted was wrong for every python-can release this driver has ever supported, and the new try/except is a harmless no-op on 3.x.


Premises re-measured this run, at base 622229f vs head 54a22990ef on python-can 4.6.1 (virtual backend):

probe base 622229f head 54a22990ef
driver.receive(0.01) 10.0001 s 0.0101 s
send_frame() then receive() raised NotImplementedError OK
peer actually received the frame yes yes
driver.receive(None) ValueError, swallowed, returned in 4.4 ms blocks (correct)

The 4.3.1 → 4.4.0 boundary was re-bisected from installed sources: 4.3.1 is a docstring-only no-op (bus.py:451), 4.4.0 raises (bus.py:458). One small extra in the fix's favour: can.CanInterfaceNotImplementedError is a subclass of NotImplementedError, so your guard covers that spelling as well. The guard costs 0.110 µs per frame — measured, and negligible.

Nothing in the earlier comment's other notes changed: the bustype= deprecation at :44, the string version comparison at :87, and the absent test coverage for this driver all stand as written.

@tridge
tridge merged commit 092d09c into dronecan:master Sep 17, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants