Sycamore

Sycamore

Synthetic Characterization for Evaluating Genomics Visualization Retrieval

Citation

Preprint: Huyen N. Nguyen, Astrid van den Brandt and N. Gehlenborg, "Sycamore: Characterizing Synthetic Personas for Evaluating Genomics Visualization Retrieval," arXiv Preprint, doi: 10.48550/arXiv.2605.08630.

@article{nguyen2026sycamore,
  author={Nguyen, Huyen N. and van den Brandt, Astrid and Gehlenborg, Nils},
  journal={arXiv Preprint}, 
  title={Sycamore: Characterizing Synthetic Personas for Evaluating Genomics Visualization Retrieval}, 
  year={2026},
  doi={10.48550/arXiv.2605.08630},
  url={https://arxiv.org/abs/2605.08630}}

A three-condition probe of how synthetic personas evaluate the Geranium retrieval system, built on top of a persona-generator engine that produces PersonaCite-grounded synthetic personas of genomics researchers.

Viewer

User interface below. For an interactive demo, go to: https://www.youtube.com/watch?v=h_Qm7L_C4CA.

Formative pass: vision + held-out test-suite stimuli

Sycamore runs as a formative pass — a cheap synthetic pre-screen that cleans protocol/interface issues and generates hypotheses before scarce expert time is spent (Munzner's abstraction-level validation, done early rather than last). Two capabilities make the pass meaningful:

1. The evaluator sees charts and reads code (vision). Each retrieved result's rendered PNG is shown to a multimodal model, and the full Gosling spec is included (not a 200-char preview), so the evaluator can judge the encoding, not just text. --vision/--no-vision, --max-images N, --spec-chars N.

2. Queries are held-out and drawn from Geranium's test suite. Point --test-suite at a Geranium checkout to run round-based exploration. Each round samples one held-out item (not in Geranium's index) and expresses the SAME information need three ways:

Modality	Query content
Image	the item's real rendered PNG (a held-out reference chart)
Spec	the item's real Gosling spec JSON
Text	each persona writes its own short, under-specified query from the alt-text (and the reference image, if vision is on) — so the query carries persona intent. Use --shared-text-query to instead reuse one cached question per item across all evaluators (RQ1 reproducibility).

Because the item is held out, retrieval must return genuine neighbours (not the query echoed back), so the evaluator reacts to real retrieval behaviour. Tying all three modalities to one item keeps modality preference interpretable and the whole slate reproducible from --seed.

Every query logs a hypothesis unit for later human validation: the information need (and whether it was persona-written or shared, via query_source), the held-out stimulus_id and chart_type, what retrieval returned (retrieved_top1, score), and the evaluator's reaction — all in the per-evaluator JSON transcript.

# Formative pass: held-out stimuli + vision, cheapest multimodal model
python -m sycamore.cli run --condition both --test-suite ../geranium \
    --vision --cheap --seed 0

# Text-only baseline (ablate vision) for comparison
python -m sycamore.cli run --condition both --test-suite ../geranium --no-vision --cheap

# Legacy behaviour (no test suite): gallery-matched Image/Spec queries
python -m sycamore.cli run --condition both --cheap

--cheap uses the cheapest multimodal model on each provider (gpt-5-nano / claude-haiku-4-5, both accept image input); synthesized text questions are cached under --out so reruns are free and deterministic. Start with --queries-per-modality 1 --k 3 to check spend.

Run Geranium first

Sycamore is a wrapper that drives a running Geranium server. You must clone and start Geranium before anything in Sycamore will work.

# ---- Terminal 1: Geranium ----
git clone https://github.com/gosling-lang/geranium     # if you haven't already
cd geranium

python -m venv .venv
source .venv/bin/activate                              # Windows: .venv\Scripts\activate
pip install -r requirements.txt                        

cd server
python app.py                                          # serves on http://localhost:5001
# leave running

Leave that running in its own terminal. Sycamore will issue search queries against it.

Live session viewer 📺

A streaming UI for watching one evaluator drive Geranium turn by turn:

# ---- Terminal 2: Sycamore one-time setup (skip if already done) ----
cd /path/to/sycamore

python -m venv .venv
source .venv/bin/activate                              # Windows: .venv\Scripts\activate
pip install -r requirements.txt

echo "ANTHROPIC_API_KEY=sk-..." > .env
echo "LLM_PROVIDER=anthropic"   >> .env

python step1_parse_personas.py                         # -> data/personas.json
python step2_parse_evidence.py                         # -> data/evidence.json
python step3_retrieve.py --rebuild                     # -> data/embeddings.npy

# ----  Launch the viewer ----
uvicorn sycamore.interface.app:app --reload --port 8001

Open your browser at http://localhost:8001 to run Sycamore live session. Pick a condition and persona, point at your Geranium URL, click Start. Cited interview excerpts render inline; abstention turns are tagged.

⚠️ Notes

• Use a separate venv per project. Geranium and Sycamore have different dependency trees (Geranium ships its own pinned versions of torch, open_clip, etc.) — keeping their venvs separate avoids version conflicts. The .venv directories should not be checked into either repo's git.
• For new shells, just reactivate. Once each venv exists, returning to that project later is just cd /path/to/project && source .venv/bin/activate. No reinstall needed unless requirements.txt changes.

What it does

For each synthetic evaluator, Sycamore replays the published Geranium user-study protocol turn by turn against the real Geranium server:

Phase	Synthetic-evaluator equivalent
Workflow description	One open-ended turn. Grounded answers cite retrieved interview excerpts; ungrounded answers come from generic LLM priors.
Tool description	The evaluator reads a textual description of Geranium and reacts.
Hands-on exploration	For each of Text, Image, Spec: the evaluator generates queries, the runner submits them to Geranium, the evaluator reacts to the returned top-k triplets.
Closing	1–3 modality ranking with rationale, plus an overall assessment.

Per-evaluator transcripts are written to data/sycamore_outputs/records/<id>.json; a run-level summary with modality aggregates and a theme-alignment scaffold goes to data/sycamore_outputs/summary.{json,md}.

Conditions

• Ungrounded: N=7 evaluators (configurable) from a generic genomics-researcher prompt; no retrieval, no validation.
• Grounded: the 7 evaluators defined in step8_evaluators.EVALUATORS (1 Biologist, 2 Computational Biologists, 2 Bioinformaticians, 2 Software Engineers). Each turn retrieves persona-scoped excerpts, filters them with a fast model, and generates a first-person response with Sources: (P_, P_) citations. If retrieval similarity falls below MIN_SIMILARITY=0.25, the evaluator abstains.

CLI

python -m sycamore.cli run \
    --condition {ungrounded|grounded|both} \
    [--n 7]                          # ungrounded evaluator count
    [--queries-per-modality 3]
    [--k 5]                          # top-k results per Geranium query
    [--geranium-url http://localhost:5001]
    [--only CB1,SE2]                 # restrict grounded run to specific abbrs
    [--out data/sycamore_outputs]

Notes & limitations

• Image and Spec queries: with --test-suite, they are real held-out items from Geranium's test suite (recommended); without it, they fall back to matching the evaluator's description to the closest gallery item.
• Run-to-run variance per persona is left to the researcher (re-run with different sampling settings).
• The expert reference condition is loaded as a JSON artefact; Sycamore does not regenerate it.