Skip to content

Backend: Optimize leaderboard query with two-stage fetch to reduce payload #5087

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
akanshajain231999 wants to merge 4 commits into
base: master
Choose a base branch
from
Open

Backend: Optimize leaderboard query with two-stage fetch to reduce payload #5087

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
cb2ddd8
Backend: Optimize leaderboard query by splitting into two-stage fetch
akanshajain231999 May 20, 2026
421cc52
Backend: Fix CI failures for leaderboard two-stage fetch
akanshajain231999 May 20, 2026
a0706d2
Merge branch 'master' into sentry_fix4
akanshajain231999 May 21, 2026
9e33e6c
Retrigger CI
akanshajain231999 May 21, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
185 changes: 108 additions & 77 deletions apps/jobs/utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -426,70 +426,27 @@
submission__is_public=True
)

if challenge_phase_split.show_execution_time:
time_diff_expression = ExpressionWrapper(
F("submission__completed_at") - F("submission__started_at"),
output_field=fields.DurationField(),
)
leaderboard_data = leaderboard_data.annotate(
filtering_score=RawSQL(
"result->>%s", (default_order_by,), output_field=FloatField()
),
filtering_error=RawSQL(
"error->>%s",
("error_{0}".format(default_order_by),),
output_field=FloatField(),
),
submission__execution_time=time_diff_expression,
).values(
"id",
"submission__participant_team",
"submission__participant_team__team_name",
"submission__participant_team__team_url",
"submission__is_baseline",
"submission__is_public",
"challenge_phase_split",
"result",
"error",
"filtering_score",
"filtering_error",
"leaderboard__schema",
"submission__submitted_at",
"submission__method_name",
"submission__id",
"submission__submission_metadata",
"submission__execution_time",
"submission__is_verified_by_host",
)
else:
leaderboard_data = leaderboard_data.annotate(
filtering_score=RawSQL(
"result->>%s", (default_order_by,), output_field=FloatField()
),
filtering_error=RawSQL(
"error->>%s",
("error_{0}".format(default_order_by),),
output_field=FloatField(),
),
).values(
"id",
"submission__participant_team",
"submission__participant_team__team_name",
"submission__participant_team__team_url",
"submission__is_baseline",
"submission__is_public",
"challenge_phase_split",
"result",
"error",
"filtering_score",
"filtering_error",
"leaderboard__schema",
"submission__submitted_at",
"submission__method_name",
"submission__id",
"submission__submission_metadata",
"submission__is_verified_by_host",
)
# Stage 1: fetch a lightweight projection (no heavy JSON columns) for
# sort + dedup. This avoids transferring result/error/submission_metadata
# for rows that will be discarded by per-team deduplication.
leaderboard_data_light = leaderboard_data.annotate(
filtering_score=RawSQL(
"result->>%s", (default_order_by,), output_field=FloatField()
),
filtering_error=RawSQL(
"error->>%s",
("error_{0}".format(default_order_by),),
output_field=FloatField(),
),
).values(
"id",
"submission__participant_team",
"submission__participant_team__team_name",
"submission__is_baseline",
"error",
"filtering_score",
"filtering_error",
)
Comment on lines +432 to +449

@coderabbitai coderabbitai Bot May 26, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Locate and print the relevant section in apps/jobs/utils.py
FILE="apps/jobs/utils.py"
echo "== Lines around 420-620 in $FILE =="
sed -n '400,620p' "$FILE" | nl -ba | sed -n '1,240p'

echo
echo "== Extract stage-1 light projection (.annotate/.values) =="
python3 - <<'PY'
import re, itertools
path="apps/jobs/utils.py"
with open(path,'r',encoding='utf-8') as f:
    lines=f.readlines()

start=0
for i,l in enumerate(lines):
    if "leaderboard_data_light = leaderboard_data.annotate" in l:
        start=i-5
        break
else:
    raise SystemExit("Could not find leaderboard_data_light annotate")

end=start+200
for idx in range(start, min(end,len(lines))):
    print(f"{idx+1:4d}: {lines[idx].rstrip()}")
PY

echo
echo "== Check whether stage-2 uses base manager or the filtered leaderboard_data queryset =="
rg -n "LeaderboardData\.objects\.filter|leaderboard_data\.filter\(" "$FILE" || true
rg -n "retained_ids|leaderboard_data_light|leaderboard_data_heavy|leaderboard_data =" "$FILE" || true

echo
echo "== Print the stage-2 heavy fetch block using retained_ids =="
python3 - <<'PY'
import re
path="apps/jobs/utils.py"
with open(path,'r',encoding='utf-8') as f:
    s=f.read()

m=re.search(r"retained_ids.*?LeaderboardData\.objects\.filter.*?(\.values|\.annotate|\))", s, flags=re.S)
if not m:
    # fallback: print around retained_ids occurrence
    lines=s.splitlines()
    for i,l in enumerate(lines):
        if "retained_ids" in l:
            a=max(0,i-40); b=min(len(lines),i+160)
            for j in range(a,b):
                print(f"{j+1:4d}: {lines[j]}")
            break
    else:
        raise SystemExit("Could not find retained_ids")
else:
    # show broader context around match start
    start=s[:m.start()].count("\n")
    lines=s.splitlines()
    a=max(0,start-40); b=min(len(lines), start+220)
    for j in range(a,b):
        print(f"{j+1:4d}: {lines[j]}")
PY

echo
echo "== Check apps/challenges/models.py for LeaderboardData.error JSONField =="
rg -n "class LeaderboardData|error\s*=\s*models\.JSONField" apps/challenges/models.py
python3 - <<'PY'
import re
path="apps/challenges/models.py"
with open(path,'r',encoding='utf-8') as f:
    lines=f.readlines()
for i,l in enumerate(lines):
    if re.search(r"error\s*=\s*models\.JSONField", l):
        a=max(0,i-10); b=min(len(lines), i+20)
        for j in range(a,b):
            print(f"{j+1:4d}: {lines[j].rstrip()}")
        break
else:
    raise SystemExit("Could not find error JSONField")
PY

Repository: Cloud-CV/EvalAI

Length of output: 169

🏁 Script executed:

#!/bin/bash
set -euo pipefail
# run the script

Repository: Cloud-CV/EvalAI

Length of output: 41

🏁 Script executed:

#!/bin/bash
set -euo pipefail

FILE="apps/jobs/utils.py"

echo "== Excerpt apps/jobs/utils.py:420-620 =="
python3 - <<'PY'
path="apps/jobs/utils.py"
lo,hi=420,620
with open(path,'r',encoding='utf-8') as f:
    for i,line in enumerate(f, start=1):
        if lo <= i <= hi:
            print(f"{i:4d}: {line.rstrip()}")
        if i > hi:
            break
PY

echo
echo "== Find stage-2 heavy fetch block around retained_ids =="
python3 - <<'PY'
import re
path="apps/jobs/utils.py"
s=open(path,'r',encoding='utf-8').read().splitlines()
# locate first occurrence of retained_ids
idx=None
for i,l in enumerate(s):
    if "retained_ids" in l:
        idx=i
        break
if idx is None:
    raise SystemExit("No retained_ids found")
start=max(0,idx-60)
end=min(len(s),idx+220)
for j in range(start,end):
    print(f"{j+1:4d}: {s[j]}")
PY

echo
echo "== Quick searches for key query sources =="
rg -n "LeaderboardData\.objects\.filter|leaderboard_data\.filter|leaderboard_data_light|leaderboard_data_heavy|retained_ids" "$FILE" || true

echo
echo "== Check apps/challenges/models.py for LeaderboardData.error JSONField =="
python3 - <<'PY'
import re
path="apps/challenges/models.py"
s=open(path,'r',encoding='utf-8').read().splitlines()
# find line containing "error = models.JSONField"
for i,l in enumerate(s):
    if re.search(r"\berror\b\s*=\s*models\.JSONField", l):
        start=max(0,i-15); end=min(len(s),i+15)
        for j in range(start,end):
            print(f"{j+1:4d}: {s[j]}")
        break
else:
    raise SystemExit("Could not find LeaderboardData.error = models.JSONField")
PY

Repository: Cloud-CV/EvalAI

Length of output: 19990

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== Excerpt apps/jobs/utils.py:330-460 (filters before stage 1) =="
python3 - <<'PY'
path="apps/jobs/utils.py"
lo,hi=330,460
with open(path,'r',encoding='utf-8') as f:
    for i,line in enumerate(f, start=1):
        if lo <= i <= hi:
            print(f"{i:4d}: {line.rstrip()}")
        if i > hi:
            break
PY

echo
echo "== Locate LeaderboardData model definition and 'error' field in apps/challenges/models.py =="
python3 - <<'PY'
import re
path="apps/challenges/models.py"
lines=open(path,'r',encoding='utf-8').read().splitlines()

# find class LeaderboardData
cls=None
for i,l in enumerate(lines):
    if re.search(r'^class\s+LeaderboardData\b', l):
        cls=i
        break
if cls is None:
    raise SystemExit("class LeaderboardData not found")

start=cls
end=min(len(lines), cls+220)
for i in range(start,end):
    if 'error' in lines[i]:
        pass

print(f"-- class LeaderboardData starts at line {cls+1} --")
for i in range(start, min(end, start+220)):
    print(f"{i+1:4d}: {lines[i].rstrip()}")
PY

echo
echo "== Search for JSONField usage with 'error' near it =="
rg -n "error" apps/challenges/models.py || true
rg -n "JSONField" apps/challenges/models.py || true

Repository: Cloud-CV/EvalAI

Length of output: 15273

🏁 Script executed:

#!/bin/bash
set -euo pipefail
FILE="apps/jobs/utils.py"

echo "== Top of apps/jobs/utils.py (imports) =="
python3 - <<'PY'
path="apps/jobs/utils.py"
with open(path,'r',encoding='utf-8') as f:
    for i in range(1,120):
        line=f.readline()
        if not line: break
        print(f"{i:4d}: {line.rstrip()}")
PY

echo
echo "== Search for Coalesce import/usages =="
rg -n "Coalesce" "$FILE" || true

Repository: Cloud-CV/EvalAI

Length of output: 5418

Fix stage-1 projection to not fetch full error JSON and ensure stage-2 reapplies stage-1 filters

  • calculate_distinct_sorted_leaderboard_data stage 1 includes "error" in .values(...), contradicting the “no heavy JSON columns” goal; this fetches the full LeaderboardData.error JSON for every candidate row.
  • Stage 2 starts from LeaderboardData.objects.filter(id__in=retained_ids) and does not reapply the stage-1 filters (including submission__is_public), so visibility/status constraints are only guaranteed via TOCTOU luck.
Suggested change 
@@
     leaderboard_data_light = leaderboard_data.annotate(
         filtering_score=RawSQL(
             "result->>%s", (default_order_by,), output_field=FloatField()
         ),
         filtering_error=RawSQL(
             "error->>%s",
             ("error_{0}".format(default_order_by),),
             output_field=FloatField(),
         ),
     ).values(
         "id",
         "submission__participant_team",
         "submission__participant_team__team_name",
         "submission__is_baseline",
-        "error",
         "filtering_score",
         "filtering_error",
     )
@@
     for leaderboard_item in leaderboard_data_light:
@@
-        if leaderboard_item["error"] is None:
+        if leaderboard_item["filtering_error"] is None:
             leaderboard_item["filtering_error"] = 0
         if leaderboard_item["filtering_score"] is None:
             leaderboard_item["filtering_score"] = 0
@@
     # Stage 2: fetch full row data (with heavy JSON columns) only for the
     # retained leaderboard rows, then reapply the order from stage 1.
     retained_ids = [item["id"] for item in retained_light]
-    heavy_qs = LeaderboardData.objects.filter(id__in=retained_ids).annotate(
+    heavy_qs = leaderboard_data.filter(id__in=retained_ids).annotate(
         filtering_score=RawSQL(
             "result->>%s", (default_order_by,), output_field=FloatField()
         ),
         filtering_error=RawSQL(
             "error->>%s",
             ("error_{0}".format(default_order_by),),
             output_field=FloatField(),
         ),
     )
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@apps/jobs/utils.py` around lines 432 - 449, In
calculate_distinct_sorted_leaderboard_data change the stage-1 projection
(leaderboard_data_light) to stop including the full "error" JSON—only include
the extracted filtering_error (from RawSQL "error->>%s") and other lightweight
fields—so you don't fetch heavy JSON per row; then ensure stage-2, where you
currently do LeaderboardData.objects.filter(id__in=retained_ids), re-applies the
same stage-1 filters (e.g. submission__is_public, any other visibility/status
filters used when building leaderboard_data) before returning final rows so the
final result set is constrained identically to stage-1 and avoids TOCTOU issues.


all_banned_participant_team = set()
all_banned_email_ids_set = (
Expand All @@ -498,15 +455,12 @@

# Apply query limit to prevent slow queries on popular challenges
max_limit = getattr(settings, "MAX_LEADERBOARD_QUERY_LIMIT", 10000)
leaderboard_data = leaderboard_data[:max_limit]

# Convert to list to allow multiple iterations
leaderboard_data = list(leaderboard_data)
leaderboard_data_light = list(leaderboard_data_light[:max_limit])

# Prefetch all participant teams and their participants' emails in bulk
# (fixes N+1 query)
unique_team_ids = set(
item["submission__participant_team"] for item in leaderboard_data
item["submission__participant_team"] for item in leaderboard_data_light
)
participant_teams = ParticipantTeam.objects.filter(
id__in=unique_team_ids
Expand All @@ -517,7 +471,7 @@
for team in participant_teams
}

for leaderboard_item in leaderboard_data:
for leaderboard_item in leaderboard_data_light:
participant_team_id = leaderboard_item["submission__participant_team"]
all_participants_email_ids = team_emails_lookup.get(
participant_team_id, []
Expand All @@ -527,21 +481,23 @@
all_banned_participant_team.add(participant_team_id)
break
if leaderboard_item["error"] is None:
leaderboard_item.update(filtering_error=0)
leaderboard_item["filtering_error"] = 0
if leaderboard_item["filtering_score"] is None:
leaderboard_item.update(filtering_score=0)
leaderboard_item["filtering_score"] = 0

Check warning on line 486 in apps/jobs/utils.py

View check run for this annotation

Codecov / codecov/patch

apps/jobs/utils.py#L486 

Added line #L486 was not covered by tests

if challenge_phase_split.show_leaderboard_by_latest_submission:
sorted_leaderboard_data = leaderboard_data
sorted_leaderboard_data = leaderboard_data_light
else:
sorted_leaderboard_data = sorted(
leaderboard_data,
leaderboard_data_light,
key=lambda k: (
float(k["filtering_score"]),
float(-k["filtering_error"]),
),
reverse=True if is_leaderboard_order_descending else False,
)
distinct_sorted_leaderboard_data = []

retained_light = []
team_list = set()
for data in sorted_leaderboard_data:
if (
Expand All @@ -551,11 +507,86 @@
):
continue
elif data["submission__is_baseline"] is True:
distinct_sorted_leaderboard_data.append(data)
retained_light.append(data)
else:
distinct_sorted_leaderboard_data.append(data)
retained_light.append(data)
team_list.add(data["submission__participant_team__team_name"])

# Stage 2: fetch full row data (with heavy JSON columns) only for the
# retained leaderboard rows, then reapply the order from stage 1.
retained_ids = [item["id"] for item in retained_light]
heavy_qs = LeaderboardData.objects.filter(id__in=retained_ids).annotate(
filtering_score=RawSQL(
"result->>%s", (default_order_by,), output_field=FloatField()
),
filtering_error=RawSQL(
"error->>%s",
("error_{0}".format(default_order_by),),
output_field=FloatField(),
),
)
if challenge_phase_split.show_execution_time:
time_diff_expression = ExpressionWrapper(
F("submission__completed_at") - F("submission__started_at"),
output_field=fields.DurationField(),
)
heavy_qs = heavy_qs.annotate(
submission__execution_time=time_diff_expression,
).values(
"id",
"submission__participant_team",
"submission__participant_team__team_name",
"submission__participant_team__team_url",
"submission__is_baseline",
"submission__is_public",
"challenge_phase_split",
"result",
"error",
"filtering_score",
"filtering_error",
"leaderboard__schema",
"submission__submitted_at",
"submission__method_name",
"submission__id",
"submission__submission_metadata",
"submission__execution_time",
"submission__is_verified_by_host",
)
else:
heavy_qs = heavy_qs.values(
"id",
"submission__participant_team",
"submission__participant_team__team_name",
"submission__participant_team__team_url",
"submission__is_baseline",
"submission__is_public",
"challenge_phase_split",
"result",
"error",
"filtering_score",
"filtering_error",
"leaderboard__schema",
"submission__submitted_at",
"submission__method_name",
"submission__id",
"submission__submission_metadata",
"submission__is_verified_by_host",
)
Comment on lines +517 to +574

@coderabbitai coderabbitai Bot May 26, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Locate and show the relevant code block
FILE="apps/jobs/utils.py"
python - <<'PY'
import os, itertools
path="apps/jobs/utils.py"
start=480
end=640
with open(path,'r',encoding='utf-8') as f:
    for i,line in enumerate(f, start=1):
        if start <= i <= end:
            print(f"{i:4d}:{line.rstrip()}")
        if i > end:
            break
PY

# Also capture where the stage-1 queryset is defined (look for likely variable names)
rg -n "leaderboard_data|retained_light|stage.?1|is_disabled|host.*exclude|public" apps/jobs/utils.py || true

# Show definition of any queryset variable used in the snippet (e.g., leaderboard_data)
rg -n "leaderboard_data\s*=" apps/jobs/utils.py || true

Repository: Cloud-CV/EvalAI

Length of output: 9444

🏁 Script executed:

#!/bin/bash
set -euo pipefail
FILE="apps/jobs/utils.py"

python - <<'PY'
path="apps/jobs/utils.py"
with open(path,'r',encoding='utf-8') as f:
    for i,line in enumerate(f, start=1):
        if 360 <= i <= 470:
            print(f"{i:4d}:{line.rstrip()}")
PY

rg -n "only_public_entries|exclude\(|is_disabled|is_public|host|flag|banned|all_banned|participant" apps/jobs/utils.py

Repository: Cloud-CV/EvalAI

Length of output: 9110

Keep stage-2 fetch scoped to the stage-1 filtered leaderboard_data queryset
Stage 1 builds leaderboard_data with is_disabled=False, host exclusion, submission__is_flagged=False, valid submission statuses, challenge_phase_split=..., and (when only_public_entries) submission__is_public=True; stage 2 re-fetches from LeaderboardData.objects by id__in=retained_ids without reapplying those predicates, so DB changes between queries can reintroduce entries that should no longer appear (the code only skips rows that disappeared, not rows whose eligibility/visibility changed).

Suggested change 
-retained_ids = [item["id"] for item in retained_light]
-heavy_qs = LeaderboardData.objects.filter(id__in=retained_ids).annotate(
+retained_ids = [item["id"] for item in retained_light]
+heavy_qs = leaderboard_data.filter(id__in=retained_ids).annotate(
     filtering_score=RawSQL(
         "result->>%s", (default_order_by,), output_field=FloatField()
     ),
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@apps/jobs/utils.py` around lines 517 - 574, The second-stage fetch re-queries
LeaderboardData via LeaderboardData.objects.filter(id__in=retained_ids) which
loses the stage-1 predicates (the original leaderboard_data queryset) so rows
that became ineligible between queries can be reintroduced; fix by basing
heavy_qs on the stage-1 queryset (e.g., use
leaderboard_data.filter(id__in=retained_ids) or chain .filter(...) onto the
original leaderboard_data) before calling .annotate(...) and .values(...),
ensuring retained_ids and retained_light are still used but the original
visibility/eligibility filters are preserved.


heavy_by_id = {row["id"]: row for row in heavy_qs}

distinct_sorted_leaderboard_data = []
for light_item in retained_light:
full = heavy_by_id.get(light_item["id"])
if full is None:
# Row removed between stage 1 and stage 2; skip.
continue

Check warning on line 583 in apps/jobs/utils.py

View check run for this annotation

Codecov / codecov/patch

apps/jobs/utils.py#L583 

Added line #L583 was not covered by tests
if full["error"] is None:
full["filtering_error"] = 0
if full["filtering_score"] is None:
full["filtering_score"] = 0

Check warning on line 587 in apps/jobs/utils.py

View check run for this annotation

Codecov / codecov/patch

apps/jobs/utils.py#L587 

Added line #L587 was not covered by tests
distinct_sorted_leaderboard_data.append(full)

leaderboard_labels = challenge_phase_split.leaderboard.schema["labels"]
show_scores = challenge_phase_split.show_scores_on_leaderboard
for item in distinct_sorted_leaderboard_data:
Expand Down
Loading
Loading