fix(vector-core): harden cosine similarity — input validation, zero-magnitude handling, AVX2 parity

[Copilot]

Description

cosine_similarity silently returned Ok(-1.0) for zero-magnitude vectors instead of erroring, and the AVX2 path had no validation at all (wrong return type, no magnitude/NaN guards). Empty vectors could be inserted into the HNSW index, propagating NaN through downstream distance calculations.

vector_distance.rs

• Empty vectors → Err(InvalidVectorData) (was: proceed to divide-by-zero)
• Zero/near-zero magnitude → Err(InvalidVectorData) via f64::EPSILON threshold (was: Ok(-1.0))
• NaN/Infinity guard on computed similarity
• Removed stale println!("mis-match in vector dimensions!")
• cosine_similarity_avx2: changed return type f64 → Result<f64, VectorError>, added same magnitude + NaN guards as scalar path

// Before: zero-magnitude silently returns misleading value
cosine_similarity(&[0.0, 0.0], &[1.0, 2.0]) // => Ok(-1.0)

// After: properly errors
cosine_similarity(&[0.0, 0.0], &[1.0, 2.0]) // => Err(InvalidVectorData)

vector_core.rs

• Empty vector validation in insert() and search() entry points
• Fixed get_all_vectors key length check: prefix + 16 → prefix + 16 + 8 (id + level)
• filter.is_none() || filter.unwrap()... → filter.as_ref().map_or(true, ...)
• Fixed typo "emtpy" → "empty" (2 occurrences)
• Removed dead commented-out code in select_neighbors

Tests — 12 new edge case tests

• Zero, near-zero, empty, both-empty, one-empty, dimension mismatch → error
• Identical, opposite, orthogonal, single-element, 1024-d vectors → correct results
• Updated test_hvector_distance_max to use opposite vectors (was zero-magnitude)
• Updated upsert test to expect error on empty vector data

Related Issues

Checklist when merging to main

• No compiler warnings (if applicable)
• Code is formatted with rustfmt
• No useless or dead code (if applicable)
• Code is easy to understand
• Doc comments are used for all functions, enums, structs, and fields (where appropriate)
• All tests pass
• Performance has not regressed (assuming change was not to fix a bug)
• Version number has been updated in helix-cli/Cargo.toml and helixdb/Cargo.toml

Additional Notes

The Cleanup artifacts CI job fails with HTTP 403 — this is a workflow permissions issue in the repo infrastructure, unrelated to these code changes.

4 files changed, +125/−42. All 1411 lib tests pass.

---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.