12789/feat/AB Testing by Sadashii · Pull Request #12788 · internetarchive/openlibrary

Sadashii · 2026-05-21T20:10:05Z

feat - This PR adds a deterministic client-and-server side A/B testing architecture.

Technical

Deterministic Hashing: User variant allocation is computed dynamically using an MD5 hash of the experiment name concatenated with the user's session identifier (cookie) or IP address fallback. This removes any database dependency or network overhead for bucketing.
Unified Backend State:
- For web.py, the ExperimentsProcessor populates web.ctx["experiments"] and infogami.utils.context.context["experiments"] (making variables directly accessible to Templetor templates/macros).
- For FastAPI, the ABTestingMiddleware registers variants on the ASGI request.state.experiments.
Global JS Availability: Experiments are serialized to the global client scope (window.OL_EXPERIMENTS) during page load. The helper getExperiment is exposed on window and as an ES module for Vue, Lit, and jQuery components to consume synchronously.
URL Overrides: One can override variant assignment on demand via query parameters in the ?experiment_[experiment_name]=[variant_name] format.

Testing

Run Backend Unit Tests:

docker compose run --rm home pytest openlibrary/tests/core/test_experiments.py

Verify URL Overrides:
- Navigate to the homepage with an override parameter: http://localhost:8080/?experiment_AB_Testing=b
- Open your browser's developer console and inspect window.OL_EXPERIMENTS (or call window.getExperiment('AB_Testing')) to verify it matches your query override.

Screenshot

No visual UI updates are introduced; this is a behavioral framework change. Manual verification of the output via browser developer console:

> window.OL_EXPERIMENTS
< { AB_Testing: "b" }

Stakeholders

Lead / Reviewers: @mekarpeles

…name]=[group] format

for more information, see https://pre-commit.ci

mekarpeles · 2026-05-21T20:12:30Z

Thank you for this contribution, @Sadashii!

Copilot has been assigned for an initial review.

A reviewer must first be assigned to this PR. There are currently 71 open PRs of equal or higher priority ahead of yours in the queue.

Possible improvements for this PR

No linked issue: This PR introduces a new feature but does not reference a GitHub issue. Adding a #NNN reference to an existing issue (or opening one) helps maintainers trace the motivation, confirm roadmap alignment, and prioritize review.

PR triage checklist (maintainers / Pam)

PR description — not empty; explains what the change does and how to verify it
References an issue — PR body contains a #NNN reference
- Linked issue is triaged — has a Priority: * label (not just Needs: Triage)
- Linked issue is assigned — has at least one assignee
Commit history clean — no WIP/fixup/conflict noise; commit messages are meaningful
CI passing — no failing check-runs
Test cases present — if the change touches substantive logic, test coverage exists or is explained
Proof of testing — PR body includes a description of what was tested, a screenshot, or a video

Note

This comment was automatically generated by Pam, Open Library's Project AI Manager, on behalf of @mekarpeles. Pam is designed to provide status visibility, perform basic project management functions and relevant codebase research, and provide actionable feedback so contributors aren't left waiting.

Copilot

Pull request overview

Adds a deterministic A/B testing framework that assigns experiment variants consistently on both legacy web.py and FastAPI paths, and exposes the resulting assignments to templates and client-side code.

Changes:

Introduces core experiment bucketing + override support (openlibrary/core/experiments.py) with unit tests.
Injects evaluated experiments into legacy template context and serializes them into the page head as window.OL_EXPERIMENTS, plus a JS helper (window.getExperiment / ES module).
Adds FastAPI middleware to populate request.state.experiments for ASGI requests.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
openlibrary/core/experiments.py	New deterministic bucketing + override evaluation for active experiments.
openlibrary/tests/core/test_experiments.py	Adds unit tests covering fallback behavior, distribution smoke test, and overrides.
openlibrary/plugins/openlibrary/processors.py	Adds a `web.py` processor to compute experiments and inject into `web.ctx` and infogami context.
openlibrary/plugins/openlibrary/code.py	Registers the new `ExperimentsProcessor` in the legacy processor chain.
openlibrary/templates/site/head.html	Serializes computed experiments into `window.OL_EXPERIMENTS` during page load.
openlibrary/plugins/openlibrary/js/experiments.js	Adds `getExperiment()` helper and exposes it on `window`.
openlibrary/plugins/openlibrary/js/index.js	Ensures experiments helper is bundled site-wide.
openlibrary/fastapi/middleware/experiments.py	Adds FastAPI middleware to compute experiments per request.
openlibrary/asgi_app.py	Registers the new FastAPI A/B testing middleware.
openlibrary/i18n/messages.pot	Updates POT output (currently appears truncated / incomplete).

…d improve typing - Convert ABTestingMiddleware to pure ASGI middleware to reduce BaseHTTPMiddleware overhead. - Restrict and filter query overrides in both web.py and FastAPI to ignore non-experiment inputs and avoid reading POST bodies. - Update get_variant and get_user_experiments signatures to support str | None for user_identifier. - These changes are based on Copilot suggestions.

…alidation - Use built-in verify_session_cookie to prevent cookie spoofing without DB/cache lookup overhead - Use Literal for audience configuration targeting rules in ExperimentConfig - Update test cases to dynamically inject ACTIVE_EXPERIMENTS and support Audience logic - Reformat code to pass all ruff check, ruff format, and mypy pre-commit hooks Co-authored-by: Copilot Suggestions <support@copilot.github.com>

for more information, see https://pre-commit.ci

Copilot

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 6 comments.

…tion, and clean up tests

for more information, see https://pre-commit.ci

mekarpeles · 2026-05-31T18:10:50Z

 def json_encode(d, indent=0) -> str:
-    return json.dumps(d, indent=indent)
+    return json.dumps(d, indent=indent).replace("<", "\\u003c").replace(">", "\\u003e").replace("&", "\\u0026")


This seems kind of hacky.

Not sure of simpler ways, do you have any recommendations?

…mps with json_encode

Sadashii added 2 commits May 22, 2026 01:22

feat: Add client and server-side A/B testing architecture

93152c5

feat: Add URL-based overrides for active experiments in ?experiment_[…

f37803b

…name]=[group] format

Copilot AI review requested due to automatic review settings May 21, 2026 20:10

Copilot started reviewing on behalf of Sadashii May 21, 2026 20:10 View session

[pre-commit.ci] auto fixes from pre-commit.com hooks

341b3aa

for more information, see https://pre-commit.ci

mekarpeles added the Needs: Submitter Input Waiting on input from the creator of the issue/pr [managed] label May 21, 2026

mekarpeles assigned cdrini May 21, 2026

mekarpeles added the Priority: 1 Do this week, receiving emails, time sensitive, . [managed] label May 21, 2026

Copilot AI reviewed May 21, 2026

View reviewed changes

Sadashii added 2 commits May 22, 2026 02:11

feat: add authorized_only parameter to A/B testing experiments

84f1e1e

github-actions Bot removed the Needs: Submitter Input Waiting on input from the creator of the issue/pr [managed] label May 21, 2026

Sadashii changed the title ~~Feat/ab testing architecture~~ 12789/feat/AB Testing May 22, 2026

mekarpeles reviewed May 26, 2026

View reviewed changes

Comment thread openlibrary/plugins/openlibrary/processors.py Outdated

Sadashii requested a review from Copilot May 28, 2026 18:43

[pre-commit.ci] auto fixes from pre-commit.com hooks

e9e432a

for more information, see https://pre-commit.ci

Copilot AI reviewed May 28, 2026

View reviewed changes

Sadashii and others added 2 commits May 29, 2026 00:28

refactor(ab-testing): unify request evaluation, secure json serializa…

4d31692

…tion, and clean up tests

[pre-commit.ci] auto fixes from pre-commit.com hooks

8d6f6f1

for more information, see https://pre-commit.ci

mekarpeles reviewed May 31, 2026

View reviewed changes

Comment thread openlibrary/plugins/openlibrary/code.py Outdated

mekarpeles reviewed May 31, 2026

View reviewed changes

Comment thread openlibrary/plugins/openlibrary/code.py Outdated

mekarpeles reviewed May 31, 2026

View reviewed changes

Sadashii added 2 commits June 1, 2026 16:29

refactor(security): unify HTML-safe JSON serialization and replace du…

8876e68

…mps with json_encode

Merge branch 'upstream/master' into feat/ab-testing-architecture

fd51484

Uh oh!

Conversation

Sadashii commented May 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Technical

Testing

Screenshot

Stakeholders

Uh oh!

mekarpeles commented May 21, 2026

Possible improvements for this PR

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mekarpeles May 31, 2026

Choose a reason for hiding this comment

Uh oh!

Sadashii Jun 1, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Sadashii commented May 21, 2026 •

edited

Loading