feat: YOLO object detection support (v8/9/10/11/12/26, TFLite, AutoNmsCalculator)#6279
Open
luolingfengflare wants to merge 15 commits into
Open
feat: YOLO object detection support (v8/9/10/11/12/26, TFLite, AutoNmsCalculator)#6279luolingfengflare wants to merge 15 commits into
luolingfengflare wants to merge 15 commits into
Conversation
Covers v8/9/10/11/12/26 support with TFLite backend, TfLiteInferenceCalculator MODEL_METADATA enhancement, AutoNmsCalculator for zero-cost NMS bypass on end-to-end models, and YoloObjectDetectorGraph architecture. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Covers TfLiteModelMetadata proto, TfLiteInferenceCalculator enhancement, YoloTensorsToDetectionsCalculator, AutoNmsCalculator, and YoloObjectDetectorGraph with full test scaffolding. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…odel_metadata.proto
Add optional MODEL_METADATA output side packet to TfLiteInferenceCalculator that emits tensor names, shapes, types, and quantization parameters after model load and delegate initialization are complete. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…or MODEL_METADATA - Use MP_ASSERT_OK_AND_ASSIGN for GetOutputSidePacket (returns StatusOr<Packet>) - Guard metadata emit block with interpreter_ != nullptr to prevent null dereference on advanced GPU path - Add ABSL_CHECK for null tensor pointer returned by interpreter_->tensor() - Add linkstatic = 1 to metadata test target to match sibling test target Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…s comment - Fix Auto_DetectsUltralyticsFromShape: shape [7,5] resolves to BOXES_FIRST (not FEATURES_FIRST); correct data population and assert label_id==0 - Add Int8_EndToEnd_QuantizationScaleOverride test covering INT8 quantized tensor path with scale=1.0/zero_point=0 - Add MismatchedQuantizationOverride_ReturnsError test asserting Open() fails with kInvalidArgument when scale is set without zero_point - Add bbox assertions for dets[1] in FamilyB_EndToEnd_TwoDetections - Add 512-class ceiling comment in ResolveTensorLayout explaining the AUTO heuristic limit and requirement for explicit layout on >512-class models Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… models) Reads MODEL_METADATA side packet in Open(), sets skip_nms_ once (dim==6 → skip), then either forwards detections unchanged or applies greedy IoU NMS. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds the options proto defining DecodeMode/TensorLayout/PostprocessMode enums, and standalone graph utils (ExtractModelInputShape, InferDecodeMode) with unit tests covering singleton-squeeze and end-to-end vs ultralytics heuristic. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…st and DISABLED model test
…lude in YoloObjectDetectorGraph
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
TfLiteModelMetadataproto — new side packet emitted byTfLiteInferenceCalculatorafterOpen(), carrying input/output tensor shapes, types, and quantization params for downstream consumersYoloTensorsToDetectionsCalculator— decodes both Family A (Ultralyticscx/cy/w/h+ argmax, v8/9/11/12) and Family B (end-to-endx1/y1/x2/y2/score/class_id, v10-e2e/v26-e2e) with AUTO shape inference from tensor dimsAutoNmsCalculator— readsMODEL_METADATAor resolves at graph build time; zero-cost NMS pass-through for end-to-end models, greedy IoU NMS for Ultralytics modelsYoloObjectDetectorGraph— fullModelTaskGraphwiringImagePreprocessingGraph → InferenceCalculator → YoloTensorsToDetectionsCalculator → AutoNmsCalculator → DetectionProjection/TransformationArchitecture
The two YOLO output families are handled by a single
decode_modeenum (AUTO,ULTRALYTICS_DETECTION_HEAD,END_TO_END). InAUTOmode the calculator detects the family at runtime from tensor shape (dim==6 → END_TO_END). NMS is applied or skipped accordingly — no runtime overhead for models that already include NMS.Test Plan
YoloTensorsToDetectionsCalculator: 9 unit tests (Family A FF/BF, Family B, AUTO detection, INT8 quantized, error paths)AutoNmsCalculator: 5 unit tests (metadata-driven skip, metadata-driven apply, no-metadata fallback, explicit mode, empty input)YoloObjectDetectorGraphutils: 8 unit tests (InferDecodeMode, ExtractModelInputShape/OutputDims logic)testdata/— enable by placingyolov8n_float32.tfliteandyolo26n_e2e_float32.tfliteand removingDISABLED_prefixFuture PRs
InferenceRunnerFactoryMODEL_METADATA.inputs[0].shape[0]already carries BDETECTIONSoutput)🤖 Generated with Claude Code