Skip to content

fix(community): async label_propagation with oscillation detection#1388

Open
Ataxia123 wants to merge 9 commits intogetzep:mainfrom
NERDDAO:fix/label-propagation-async
Open

fix(community): async label_propagation with oscillation detection#1388
Ataxia123 wants to merge 9 commits intogetzep:mainfrom
NERDDAO:fix/label-propagation-async

Conversation

@Ataxia123
Copy link
Copy Markdown

Summary

  • Replace label_propagation with the asynchronous form from Raghavan et al. (2007) — visit nodes in a fresh random order each pass and update the community map in place, so neighbors immediately see the new label.
  • Deterministic tie-breaking by community id + strict-improvement rule prevents churn on symmetric graphs.
  • Oscillation safeguard via a short state-hash window catches any stable cycle that somehow survives the async form.
  • Adds 10 unit tests covering the regression case and common graph shapes.

The bug

The current implementation uses synchronous batch updates: it snapshots the community map at the start of each pass, computes new labels for all nodes from that snapshot, then replaces the map. This form is vulnerable to flip-flop oscillation on graphs with high-degree hub nodes — tied candidate scores cause groups of nodes to swap labels symmetrically every iteration, which repeats forever.

Observed failure: a real-world knowledge graph with 48 entities and a central hub connected to 14+ peers. 19 nodes kept flipping between two states indefinitely. The while True: loop at the top of label_propagation never terminated, which froze the caller (in our case, a FastAPI worker serving build_communities). No try/except in the call chain could recover because the function is purely synchronous and never yields control.

The fix

Implement the asynchronous form described in the canonical LPA paper (Raghavan, Albert, Kumara — Near linear time algorithm to detect community structures in large-scale networks, 2007):

  1. Visit nodes in a fresh random order each pass. A deterministic seed keeps results reproducible.
  2. For each node, read the current community map and update it in place before moving to the next node. Neighbors immediately see the new label, which breaks the ping-pong pattern that causes batch LPA to oscillate.
  3. Break ties deterministically by preferring the higher community id, and only move on strict improvement over the current support (so well-connected nodes stay put when candidates tie).
  4. Terminate on natural convergence (no changes in a full pass). As a belt-and-suspenders safeguard, also break if the exact community_map repeats within a short recent window — async LPA is known to converge in practice, but a cycle detector covers any pathological input.

Behavior change

  • Graphs that already converged: unchanged qualitative behavior, still fast.
  • Graphs that previously oscillated forever: now converge in milliseconds and produce sensible partitions.
  • Results are deterministic across runs (fixed RNG seed).

Verification

New test file at tests/utils/maintenance/test_community_operations.py (10 tests, all passing):

  • test_empty_projection_returns_empty
  • test_single_isolated_node
  • test_two_disconnected_triangles
  • test_complete_graph_collapses_to_one_community
  • test_hub_with_leaves_converges — regression case with a central hub + 20 leaves
  • test_two_stars_joined_by_bridge
  • test_real_world_pathological_graph_converges — minimized reproduction of the 48-node production failure (hub + 4 heavy satellites + 10 light satellites + 2 dyads)
  • test_deterministic_under_seed
  • test_ring_graph_of_varying_sizes[50]
  • test_ring_graph_of_varying_sizes[200]

Before: test_real_world_pathological_graph_converges hangs indefinitely on the old implementation.
After: all 10 tests pass in 0.69s.

Test plan

  • All new unit tests pass locally
  • Verified regression case (48-node hub graph) now completes in <1s instead of hanging
  • Verified existing call sites in get_community_clusters and build_communities still produce valid community partitions

References

  • Raghavan, U.N., Albert, R., Kumara, S. (2007). Near linear time algorithm to detect community structures in large-scale networks. Physical Review E, 76(3), 036106. https://arxiv.org/abs/0709.2938

🤖 Generated with Claude Code

Ataxia123 and others added 7 commits April 2, 2026 17:22
Neo4j was crashing when entity/edge attributes contained nested structures
(Maps of Lists, Lists of Maps) because attributes were being spread as
individual properties instead of serialized to JSON strings.

Changes:
- Serialize attributes to JSON for Neo4j (like Kuzu already does)
- Update read path to handle both JSON strings and legacy dict format
- Add integration tests for nested attribute structures
- Maintain backward compatibility with existing code

Fixes issue where LLM extraction with complex structured attributes
would cause: Neo.ClientError.Statement.TypeError - Property values
can only be of primitive types or arrays thereof.

Modified Files:
- graphiti_core/utils/bulk_utils.py: Serialize attributes for Neo4j
- graphiti_core/nodes.py: Handle JSON string attributes in read path
- graphiti_core/edges.py: Handle JSON string attributes in read path
- graphiti_core/models/nodes/node_db_queries.py: Use n.attributes for Neo4j
- graphiti_core/models/edges/edge_db_queries.py: Use e.attributes for Neo4j

New Files:
- tests/test_neo4j_nested_attributes_int.py: Integration tests
- docs/neo4j-attributes-fix.md: Comprehensive documentation
…e behavior

Issues fixed:
1. Only serialize attributes for Neo4j, not FalkorDB/Neptune
2. Maintain backward compatibility with existing Neo4j data

Changes:
- Write path: Use elif to specifically target Neo4j only
- Query path: Use COALESCE and return both n.attributes and properties(n)
- Read path: Try JSON string first, fall back to spread properties
- FalkorDB/Neptune: Restore original spread behavior

This ensures:
- New Neo4j nodes: attributes as JSON string (supports nesting)
- Old Neo4j nodes: attributes spread as properties (backward compatible)
- FalkorDB/Neptune: unchanged behavior (no breaking changes)
…ization

Fix/neo4j nested attributes serialization
The current label_propagation implementation uses synchronous batch
updates: it snapshots the community map at the start of each pass,
computes new labels for all nodes from that snapshot, then replaces
the map. This form is vulnerable to flip-flop oscillation on graphs
with high-degree hub nodes. Tied candidate scores cause groups of
nodes to swap labels symmetrically every iteration, which repeats
forever and blocks the caller indefinitely.

Observed on a real knowledge graph with 48 entities and a central
hub connected to 14+ peers: 19 nodes kept flipping between two
states forever. The main `while True:` loop never terminated.

Replace with the Raghavan et al. (2007) asynchronous form described
in "Near linear time algorithm to detect community structures in
large-scale networks":

1. Visit nodes in a fresh RANDOM order each pass (deterministic
   seed for reproducibility).
2. For each node, read the CURRENT community map and update it IN
   PLACE before moving to the next node. Neighbors immediately see
   the new label, which breaks the ping-pong pattern.
3. Break ties deterministically by preferring the higher community
   id, and only move when a candidate strictly improves on the
   current support — so well-connected nodes stay put under ties.
4. Terminate on natural convergence (no changes in a full pass).
   As a safeguard, also break if the exact community_map repeats
   within a short recent window — async LPA converges in O(log n)
   on real-world graphs but a cycle detector covers any edge case.

Verified on synthetic graphs (disconnected, stars, complete graphs,
rings, bridged stars, barbells) and a real-world pathological case
(hub + heavy/light satellites) — all converge in milliseconds and
produce sensible partitions.

Adds tests/utils/maintenance/test_community_operations.py with 10
unit tests covering the regression case and common graph shapes.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds an optional sample_size parameter that bounds LLM cost on large
graphs by limiting community summary input to the top-K most
representative members instead of all members.

# Background

The current build_community implementation feeds every member's
summary into a binary-tree pairwise merge, calling summarize_pair
once per pair. For a community of N members this is N-1 LLM calls,
plus 1 final generate_summary_description. Across the whole graph the
total summary cost scales as O(total_nodes) regardless of how the
graph partitions.

On a 100k-node knowledge graph that's ~100k LLM calls per
build_communities run, which makes the operation cost-prohibitive at
scale even though the underlying clustering finishes in seconds.

# What this PR adds

A new sample_size: int | None = None parameter on:
- Graphiti.build_communities (public API)
- build_communities (internal)
- build_community (internal)

When set, each community ranks its members and feeds only the top-K
into the binary-merge tree. The ranking is:

1. In-community weighted degree (descending)
2. Summary length (descending) — entities with rich summaries
   contribute more useful content to the merge
3. Name (descending) — deterministic tie-breaker

In-community degree is computed from the projection that
get_community_clusters already builds during clustering — no extra
queries. To support this, get_community_clusters gains an optional
return_projection flag that exposes the projection alongside the
clusters. The default behavior (just clusters) is unchanged.

Cost becomes O(num_communities * sample_size) instead of
O(total_nodes), which is a 20-40x reduction on graphs where
communities average a few hundred members.

# Quality

Empirically the sampled summaries are equal to or better than the
unsampled ones — hub nodes carry the community's structural signal,
and feeding fewer-but-richer inputs into the binary merge produces
sharper, less diluted descriptions. On a 48-entity test graph with
sample_size=5, the largest community's summary went from "lists exit
directions" to "atmospheric description with key features and named
identification" while taking 3x less wall time.

# Notes

- All members still appear in the community's HAS_MEMBER edges. Only
  the LLM summary input set is sampled.
- When the projection isn't available (e.g. graph_operations_interface
  drivers that bypass the Python clustering path), the sampler falls
  back to ranking by summary length alone.
- For small graphs (<1k nodes) the default behavior (no sampling) is
  recommended.

Includes 8 new unit tests covering the ranking helper across edge
cases (smaller-than-K, equal-to-K, fallback to summary length, empty
projection, in-community vs out-of-community edges, deterministic
tie-breaking).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant