Skip to content

Add Sonic Similarity plugin for audio-based track similarity#1

Closed
chrisuthe wants to merge 6 commits intotask/sonic-analysis-providerfrom
task/sonic-similarity-plugin
Closed

Add Sonic Similarity plugin for audio-based track similarity#1
chrisuthe wants to merge 6 commits intotask/sonic-analysis-providerfrom
task/sonic-similarity-plugin

Conversation

@chrisuthe
Copy link
Copy Markdown
Owner

@chrisuthe chrisuthe commented Apr 1, 2026

Sonic Similarity Plugin

This PR adds a similarity search plugin that builds a fast vector index over audio analysis data and exposes a general-purpose API for finding similar tracks.

Stacks on top of the sonic analysis provider (upstream PR music-assistant#3516).

How It Works

  1. On startup, reads all AudioAnalysisData rows from the database
  2. Assembles a 14-dimensional vector from each track's semantic fields
  3. Normalizes vectors (z-score + L2) and inserts them into a USearch HNSW index
  4. Exposes sonic_analysis/similar API for nearest-neighbor search

Vector Schema (14 dimensions)

The plugin owns the vector definition — the provider just stores semantic fields, the plugin decides how to turn them into a searchable vector.

Dims Content
0-8 9 scalar features: bpm, energy, danceability, loudness_integrated, loudness_range, brightness, harmonic_complexity, roughness, rhythmic_regularity
9-11 Circular key encoding (sin, cos) + mode (major=1, minor=0)
12-13 Time-series variance: RMS energy dynamics, spectral centroid movement

API: sonic_analysis/similar

Single endpoint that covers every use case through parameters:

Capability How
"Play something like this" item_id="42", limit=1
"Build a similar-tracks list" item_ids=["42"], limit=25, resolve=True
"Radio that drifts naturally" item_ids=[...last 5], depth=2, diversity=0.2
"Mix these vibes" item_ids=[a,b,c], blend_mode="union", limit=50
"Similar jazz from my NAS only" filter_genres=["jazz"], filter_providers=["prov_local"]
Per-group distance breakdown include_group_distances=True

Features:

  • Multi-seed blending: centroid (weighted average) or union (merge neighborhoods)
  • Recursive depth: expands results across generations for organic radio drift
  • MMR diversity: Maximal Marginal Relevance prevents results from clustering too tightly
  • Per-group distances: optional breakdown by rhythm, loudness, timbre, regularity, tonal, dynamics
  • Presets: balanced, vibe, party, genre_era, discover — each weights the 6 feature groups differently
  • Filtering: by provider instance, genre, exclude artists/tracks
  • Backfill: bulk-analyze unanalyzed library tracks from local files

Code Organization

  • vectors.py — Vector schema definition, assembly from AudioAnalysisData, normalization, corpus stats, weighted distance computation. Zero MA dependencies.
  • similarity.py — Pure math: centroid blending, union merge, MMR diversity, recursive expansion. Zero MA dependencies.
  • __init__.py — MA integration: USearch index management, API handlers, backfill, label mapping, metadata re-ranking, debug UI

Testing

  • test_vector_assembly.py — 38 tests for vector assembly, key encoding, normalization, corpus stats, weighted distance
  • test_group_distances.py — 8 tests for per-group distance breakdown
  • test_similarity.py — 17 tests for pure similarity functions (centroid, union, MMR, recursive expansion)
  • test_plugin_api.py — 25 tests for parameter validation, weight parsing, filters, backward compat

Debug UI

Browser-based debug console at /sonic_analysis/debug with WebSocket controls for searching similar tracks, triggering backfill, rebuilding the index, and tuning all parameters (depth, blend mode, diversity, weight sliders, presets).

Dependencies

@chrisuthe chrisuthe force-pushed the task/sonic-similarity-plugin branch 3 times, most recently from 8205e59 to e46a759 Compare April 2, 2026 16:02
@chrisuthe chrisuthe force-pushed the task/sonic-analysis-provider branch from 729ed8a to 3d93152 Compare April 5, 2026 15:37
@chrisuthe chrisuthe force-pushed the task/sonic-similarity-plugin branch from e46a759 to 2f5dd0a Compare April 5, 2026 15:37
@chrisuthe chrisuthe force-pushed the task/sonic-analysis-provider branch from 3d93152 to b04676e Compare April 21, 2026 15:37
@chrisuthe chrisuthe force-pushed the task/sonic-similarity-plugin branch 2 times, most recently from 1c86642 to 19a32f3 Compare April 21, 2026 15:42
@chrisuthe chrisuthe force-pushed the task/sonic-analysis-provider branch from b04676e to fa65f09 Compare April 21, 2026 16:24
@chrisuthe chrisuthe force-pushed the task/sonic-similarity-plugin branch from 19a32f3 to dfb6426 Compare April 21, 2026 16:25
…earch indexing, and debug UI

Similarity search plugin with:
- 17-dim vector assembly from AudioAnalysisData (9 required + 3 optional ML + key/mode + dynamics)
- USearch HNSW index with per-provider separate indexes
- Presets (balanced, vibe, party, genre_era, discover) with 7 feature groups including mood
- Per-group distance breakdown, MMR diversity, recursive depth expansion
- Provider comparison endpoint, analysis export with MBID/ISRC
- Debug page with weight sliders, backfill, provider switching, CSV export
- Configurable aa_provider_domain for switching between sonic/essentia analysis

Adapted API paths to upstream: mass.streams.audio_analysis.*
@chrisuthe chrisuthe force-pushed the task/sonic-analysis-provider branch from fa65f09 to 7f5d82d Compare April 21, 2026 16:28
@chrisuthe chrisuthe force-pushed the task/sonic-similarity-plugin branch from dfb6426 to 6db8ac4 Compare April 21, 2026 16:28
Upstream merged arousal field on AudioAnalysisData. Add it to
OPTIONAL_FIELDS in vectors.py alongside instrumentalness, valence,
acousticness. Mood group now covers 4 dims (9-13).

Defaults to 0.5 when absent (librosa/lite providers don't produce it).
…mpare/export/switching capabilities

The rebase copied an early version of the plugin missing:
- Audio Analysis Providers panel with config editing
- Compare Providers endpoint and debug UI
- Export analysis with MBID/ISRC, random pick, ML fields
- Provider switching with separate indexes
- AA provider config save API
- Mood/genre display in export and compare views

All features restored. API paths updated for upstream.
Arousal was in the vector (OPTIONAL_FIELDS) but missing from:
- export_fields (export API)
- compare_fields (compare providers API)
- JS EXPORT_COLS (debug page export table/CSV)
- JS compare fields (debug page compare view)
@chrisuthe chrisuthe force-pushed the task/sonic-similarity-plugin branch from 6db8ac4 to 83e77ad Compare April 21, 2026 18:07
chrisuthe added a commit that referenced this pull request Apr 28, 2026
Both bulk-read sites (_load_overlay_overrides and _rebuild_search_index)
now call mass.streams.audio_analysis.list_rows_by_domain() instead of
hitting mass.music.database directly. Per upstream rule that providers
go through controllers, not the DB.

Stacks on PR #1 (feat/sonic-analysis-provider-pr) for the controller
helpers; once PR #1 lands and this rebases onto dev, the dependency
becomes implicit.
chrisuthe added a commit that referenced this pull request Apr 28, 2026
_rebuild_index_from_database now calls
mass.streams.audio_analysis.list_rows_by_domain() instead of hitting
mass.music.database directly. Per upstream rule that providers go
through controllers, not the DB.

Stacks on PR #1 (feat/sonic-analysis-provider-pr) for the controller
helpers; once PR #1 lands and this rebases onto dev, the dependency
becomes implicit.
chrisuthe added a commit that referenced this pull request Apr 29, 2026
Both bulk-read sites (_load_overlay_overrides and _rebuild_search_index)
now call mass.streams.audio_analysis.list_rows_by_domain() instead of
hitting mass.music.database directly. Per upstream rule that providers
go through controllers, not the DB.

Stacks on PR #1 (feat/sonic-analysis-provider-pr) for the controller
helpers; once PR #1 lands and this rebases onto dev, the dependency
becomes implicit.
chrisuthe added a commit that referenced this pull request Apr 29, 2026
_rebuild_index_from_database now calls
mass.streams.audio_analysis.list_rows_by_domain() instead of hitting
mass.music.database directly. Per upstream rule that providers go
through controllers, not the DB.

Stacks on PR #1 (feat/sonic-analysis-provider-pr) for the controller
helpers; once PR #1 lands and this rebases onto dev, the dependency
becomes implicit.
chrisuthe added a commit that referenced this pull request Apr 30, 2026
Both bulk-read sites (_load_overlay_overrides and _rebuild_search_index)
now call mass.streams.audio_analysis.get_audio_analysis_rows() instead
of hitting mass.music.database directly. Per upstream rule that
providers go through controllers, not the DB.

Stacks on PR #1 (feat/sonic-analysis-provider-pr) for the controller
helpers; once PR #1 lands and this rebases onto dev, the dependency
becomes implicit.
chrisuthe added a commit that referenced this pull request Apr 30, 2026
_rebuild_index_from_database now calls
mass.streams.audio_analysis.get_audio_analysis_rows() instead of hitting
mass.music.database directly. Per upstream rule that providers go
through controllers, not the DB.

Stacks on PR #1 (feat/sonic-analysis-provider-pr) for the controller
helpers; once PR #1 lands and this rebases onto dev, the dependency
becomes implicit.
chrisuthe added a commit that referenced this pull request Apr 30, 2026
_rebuild_index_from_database now calls
mass.streams.audio_analysis.get_audio_analysis_rows() instead of hitting
mass.music.database directly. Per upstream rule that providers go
through controllers, not the DB.

Stacks on PR #1 (feat/sonic-analysis-provider-pr) for the controller
helpers; once PR #1 lands and this rebases onto dev, the dependency
becomes implicit.
chrisuthe added a commit that referenced this pull request Apr 30, 2026
Both bulk-read sites (_load_overlay_overrides and _rebuild_search_index)
now call mass.streams.audio_analysis.get_audio_analysis_rows() instead
of hitting mass.music.database directly. Per upstream rule that
providers go through controllers, not the DB.

Stacks on PR #1 (feat/sonic-analysis-provider-pr) for the controller
helpers; once PR #1 lands and this rebases onto dev, the dependency
becomes implicit.
chrisuthe added a commit that referenced this pull request Apr 30, 2026
Both bulk-read sites (_load_overlay_overrides and _rebuild_search_index)
now call mass.streams.audio_analysis.get_audio_analysis_rows() instead
of hitting mass.music.database directly. Per upstream rule that
providers go through controllers, not the DB.

Stacks on PR #1 (feat/sonic-analysis-provider-pr) for the controller
helpers; once PR #1 lands and this rebases onto dev, the dependency
becomes implicit.
@chrisuthe chrisuthe closed this May 4, 2026
@chrisuthe chrisuthe deleted the task/sonic-similarity-plugin branch May 4, 2026 19:45
chrisuthe added a commit that referenced this pull request May 4, 2026
Both bulk-read sites (_load_overlay_overrides and _rebuild_search_index)
now call mass.streams.audio_analysis.get_audio_analysis_rows() instead
of hitting mass.music.database directly. Per upstream rule that
providers go through controllers, not the DB.

Stacks on PR #1 (feat/sonic-analysis-provider-pr) for the controller
helpers; once PR #1 lands and this rebases onto dev, the dependency
becomes implicit.
chrisuthe added a commit that referenced this pull request May 4, 2026
_rebuild_index_from_database now calls
mass.streams.audio_analysis.get_audio_analysis_rows() instead of hitting
mass.music.database directly. Per upstream rule that providers go
through controllers, not the DB.

Stacks on PR #1 (feat/sonic-analysis-provider-pr) for the controller
helpers; once PR #1 lands and this rebases onto dev, the dependency
becomes implicit.
chrisuthe added a commit that referenced this pull request May 4, 2026
Both bulk-read sites (_load_overlay_overrides and _rebuild_search_index)
now call mass.streams.audio_analysis.get_audio_analysis_rows() instead
of hitting mass.music.database directly. Per upstream rule that
providers go through controllers, not the DB.

Stacks on PR #1 (feat/sonic-analysis-provider-pr) for the controller
helpers; once PR #1 lands and this rebases onto dev, the dependency
becomes implicit.
chrisuthe added a commit that referenced this pull request May 4, 2026
_rebuild_index_from_database now calls
mass.streams.audio_analysis.get_audio_analysis_rows() instead of hitting
mass.music.database directly. Per upstream rule that providers go
through controllers, not the DB.

Stacks on PR #1 (feat/sonic-analysis-provider-pr) for the controller
helpers; once PR #1 lands and this rebases onto dev, the dependency
becomes implicit.
chrisuthe added a commit that referenced this pull request May 5, 2026
Both bulk-read sites (_load_overlay_overrides and _rebuild_search_index)
now call mass.streams.audio_analysis.get_audio_analysis_rows() instead
of hitting mass.music.database directly. Per upstream rule that
providers go through controllers, not the DB.

Stacks on PR #1 (feat/sonic-analysis-provider-pr) for the controller
helpers; once PR #1 lands and this rebases onto dev, the dependency
becomes implicit.
chrisuthe added a commit that referenced this pull request May 5, 2026
_rebuild_index_from_database now calls
mass.streams.audio_analysis.get_audio_analysis_rows() instead of hitting
mass.music.database directly. Per upstream rule that providers go
through controllers, not the DB.

Stacks on PR #1 (feat/sonic-analysis-provider-pr) for the controller
helpers; once PR #1 lands and this rebases onto dev, the dependency
becomes implicit.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant