Beyond Scalar Leaderboards

Code repository for the paper Beyond Scalar Leaderboards: Adaptive Sampling for Pareto Frontier Identification in Multi-Objective LLM Arenas.

This repository brings together the synthetic experiments and realistic Arena replay experiments for identifying a multi-objective Pareto frontier from pairwise feedback. It includes Vector-Borda Empirical-Gap Elimination (VB-EGE), comparison methods, pinned experimental protocols, tests, and code for generating statistical summaries and paper figures. Datasets and generated results are not tracked.

Start here: Installation and quick start · Reproduction guide · Dataset specification · Validation record

Research problem

A scalar leaderboard aggregates performance into one number. Here each model has a vector of scores, one per objective. For model $i$ and objective $c$, its uniform-opponent Vector-Borda score is

$$b_{i,c}=\frac{1}{K-1}\sum_{j\ne i}p_{ij,c},$$

where $p_{ij,c}$ is the probability that model $i$ wins against model $j$ on objective $c$. All coordinates are maximized. The target uses strict improvement in every coordinate:

$$\mathcal P={i:\nexists j\ne i\text{ such that }b_{j,c}>b_{i,c}\text{ for every }c}.$$

Equality in even one coordinate prevents this form of dominance. This convention matters when comparing against other Pareto implementations.

VB-EGE adaptively allocates comparisons to active model-objective cells and eliminates or accepts models using empirical identification gaps. Opponents are sampled uniformly from the original roster, including models already removed from the active set. The main configuration uses $(C_{\mathrm{samp}},C_{\mathrm{thr}},C_{\log})=(2,4,4)$.

Repository structure

Beyond_Scalar_Leaderboards/
├── README.md                        Paper and code overview
├── REPRODUCIBILITY.md                Unified reproduction instructions
├── docs/                            Validation environment and checks
├── Exp_Synthetic/
│   ├── vb_ege/                      Algorithms, instances, metrics, runners
│   ├── configs/                     Six paper suites and a smoke protocol
│   ├── scripts/                     Verification and experiment runners
│   ├── tests/                       Synthetic and numerical tests
│   ├── icml_pareto_track_stop/       Exploratory BT-GLR extension
│   ├── pilot_focal_bt_mle/           Constrained BT numerical utilities
│   ├── pyproject.toml
│   └── uv.lock
└── Exp_Realistic/
    ├── arena_vbege/                 Replay, data processing, algorithms
    ├── configs/                     Arena-35, SearchArena-8, VisionArena-12
    ├── experiments/                 Tie and Davidson sensitivity analyses
    ├── tests/                       Algorithm, data, and pipeline tests
    ├── DATASETS.md                  Source revisions and cleaning rules
    ├── pyproject.toml
    └── uv.lock

The two subprojects retain separate dependency locks and execution directories. There is no root Python package: run each command from the indicated subproject.

Installation and quick start

Use Python 3.12 and uv. Both package metadata files allow Python 3.10+, but Python 3.12 is the reference environment. A CPU is sufficient; the experiments operate on simulated or logged comparisons and do not require live LLM API calls or a GPU.

Download or clone this repository, then open a terminal in its root directory.

Install and validate the synthetic subproject:

cd Exp_Synthetic
uv sync --python 3.12 --extra dev --locked
uv run --locked python -m pytest
uv run --locked python -m vb_ege.run_sweep --config configs/smoke.yaml --out results/raw/smoke.csv --seed 0
uv run --locked python -m vb_ege.summarize --raw results/raw/smoke.csv --out results/summary/smoke_summary.csv --figdir results/figures/smoke

Then validate the realistic subproject without downloading datasets:

cd ../Exp_Realistic
uv sync --python 3.12 --extra dev --locked
uv run --locked python -m pytest
uv run --locked python -m arena_vbege.cli --help

These commands work in PowerShell and POSIX shells without activating a virtual environment. Each uv sync creates a separate .venv in its subproject. The smoke configuration uses reduced constants and repetitions for implementation checks; it is not the paper's main statistical protocol.

Experiments

Synthetic experiments

Study	Configuration in Exp_Synthetic/configs/	Scheduled raw rows
Scaling in models, objectives, and gap	fixed_confidence_scaling.yaml	24,000
Heterogeneous benchmark instances	fixed_confidence_benchmarks.yaml	16,000
Confidence scaling and tails	confidence_scaling_quantile.yaml	23,200
Sampling and threshold constants	constants_calibration.yaml	19,800
Pareto frontier size	pareto_size_ablation.yaml	9,600
Conservative constants	theory_constants_sanity.yaml	400

See the synthetic overview and detailed synthetic protocol for generators, seed schedules, raw-row fields, paired comparisons, and uncertainty summaries. Scheduled counts describe the supplied configurations; see the validation record for checks performed.

Realistic Arena replay

Setting	Models	Objectives	Empirical frontier size	Dataset
Arena-35	35	5	3	lmarena-ai/arena-human-preference-140k
SearchArena-8	8	9	5	lmarena-ai/search-arena-24k
VisionArena-12	12	8	3	lmarena-ai/VisionArena-Battle

The main protocol retains decisive binary outcomes, routes comparisons using released objective metadata, selects a complete roster, and replays logged pair-objective cells with replacement. Each setting runs four methods over 100 seeds at delta=0.05. Exact source revisions, routing rules, and preparation counts are specified in DATASETS.md.

Methods and interpretation

Method or component	Role and qualification
VB-EGE	Main adaptive Vector-Borda method; VB-EGE-practical is its retained historical raw-data label.
Uniform focal Borda	Uniform focal model-objective sampling baseline.
Uniform pairwise Borda plug-in	Pair-objective sampling with Borda reconstruction.
Uniform pairwise BT-MLE	Model-based diagnostic; its empirical stopping rule is not a proved delta-correct region for arbitrary Arena cells.
Pareto BT-GLR Track-and-Stop	Exploratory synthetic extension used in supplementary experiments; no delta-correctness guarantee is claimed.
Tie and Davidson sensitivities	Alternative outcome models and empirical targets, reported separately from decisive-only experiments.

Realistic repeated-seed uncertainty is conditional on a fixed empirical log. It does not quantify uncertainty from new users, future traffic, prompts, or matchmaking. Displayed 2D/3D projections do not replace the full-dimensional target.

Data and generated artifacts

The current repository tree contains code, configurations, lockfiles, tests, and documentation. Downloaded datasets, derived sensitivity tables, raw results, summary tables, result figures, virtual environments, and caches are excluded. The three dataset links above point to their upstream Hugging Face pages. VisionArena is configured as a gated source: obtain access through its provider and authenticate locally before preparation.

Run the supplied scripts to generate results and paper figures locally; see REPRODUCIBILITY.md. Tracked files beneath results/ are limited to a directory guide and empty .gitkeep placeholders.

Citation and licensing

If you use this code, please cite the accompanying paper, Beyond Scalar Leaderboards: Adaptive Sampling for Pareto Frontier Identification in Multi-Objective LLM Arenas, and identify the repository commit used.

No project-wide software license is specified. Dataset access and use remain subject to the respective providers' terms.