Skip to content

Avoid pulling pre-2023 data into movedBoxes stat#2704

Merged
aerinsol merged 1 commit intomasterfrom
fix-pulling-old-data-into-stats
May 8, 2026
Merged

Avoid pulling pre-2023 data into movedBoxes stat#2704
aerinsol merged 1 commit intomasterfrom
fix-pulling-old-data-into-stats

Conversation

@pylipp
Copy link
Copy Markdown
Contributor

@pylipp pylipp commented May 5, 2026

https://trello.com/c/EFSdLs3F

This didn't affect data display because in the FE we filter by a date range (earliest 2023-01-01).
It reduces the querying time and request size (and hence latency) for bases with a considerable amount of pre-2023 boxes.

@pylipp
Copy link
Copy Markdown
Contributor Author

pylipp commented May 5, 2026

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adjusts the backend SQL used for the movedBoxes statistics so that, in production, all history rows considered for box reconstruction (including “Record created/deleted/undeleted” events) are constrained by the configured “earliest 2023” history.id cutoff, preventing pre-2023 history events from being pulled into the dataset.

Changes:

  • Tightened the BoxHistory CTE filter so h.id >= %s applies uniformly (including to “Record created/deleted/undeleted” change types).
  • Simplified the boolean logic by grouping the change-type checks under a single h.id >= %s guard.

@sentry
Copy link
Copy Markdown

sentry Bot commented May 5, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 76.83%. Comparing base (1003b9a) to head (db2f4ce).

Additional details and impacted files
@@           Coverage Diff           @@
##           master    #2704   +/-   ##
=======================================
  Coverage   76.83%   76.83%           
=======================================
  Files         300      300           
  Lines       22233    22233           
  Branches     2245     2245           
=======================================
  Hits        17082    17082           
  Misses       5104     5104           
  Partials       47       47           
Flag Coverage Δ
backend 99.65% <ø> (ø)
frontend 69.10% <ø> (ø)
sharedComponents 68.72% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@aerinsol
Copy link
Copy Markdown
Member

aerinsol commented May 6, 2026

@pylipp I assume that %s is a variable - is there a reason the computation date isn't hardcoded in? Or are you setting this in a central module where it's then called by different functions?

@pylipp
Copy link
Copy Markdown
Contributor Author

pylipp commented May 6, 2026

@pylipp I assume that %s is a variable - is there a reason the computation date isn't hardcoded in? Or are you setting this in a central module where it's then called by different functions?

@aerinsol yes, the query is invoked from one place which passes the cut-off history ID. Since that's not compatible for testing, this variable is only set in production.
We could probably switch it for a date-date-threshold and hardcode it but I have to check if the history.changes column is indexed

@pylipp
Copy link
Copy Markdown
Contributor Author

pylipp commented May 7, 2026

We could probably switch it for a date-date-threshold and hardcode it but I have to check if the history.changedate column is indexed

@aerinsol the changedate column is not indexed (only composite index with tablename or record_id), so I stick with the approach to filter for a variable h.id (for testing set to 0; for production, the first ID in 2023), code for reference

@pylipp pylipp requested a review from aerinsol May 7, 2026 09:00
@aerinsol aerinsol merged commit fcd0536 into master May 8, 2026
18 checks passed
@aerinsol aerinsol deleted the fix-pulling-old-data-into-stats branch May 8, 2026 15:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants