Loculus query-syntax playground

An interactive sandbox for designing the Loculus filter query language — the thing that goes in

GET /sample/details?filter=<query>&fields=…&limit=…

Build a query with the visual builder, then flip the design choices in the left-hand panel and watch the query string, the grammar, the parse behaviour and the number of matching records change.

Everything runs in the page. No build step, no dependencies, no server.

Running it

# straight from disk
open index.html          # or xdg-open / just double-click it

# or over http, if you prefer
python3 -m http.server 8000   # → http://localhost:8000

Deploying to GitHub Pages: push the repository and enable Pages (Settings → Pages → Source: GitHub Actions). .github/workflows/pages.yml publishes the repository root as-is.

Tests

node test/run.js          # serialise → parse → serialise over 16 examples × 212 dialect variants,
                          # plus semantic checks against the demo dataset
npm i jsdom && node test/ui.js   # loads index.html in jsdom and clicks through every control

What it covers

The query language itself:

metadata	country==Germany, country=in=(Germany,Switzerland), coverage>=0.9
dates	sampleCollectionDate=range=(2021-02-01,2021-04-30), half-open intervals
imprecise dates	dateSubmitted>~2024-02-15 — the "possibly" reading, for a date known only to the month
lineages	lineage=descendantOf=B.1.1.7 — alias-aware, unlike a prefix match
amino acids	aa.S.501==Y, aa.S.501=in=(Y,K), aa.S.681!=P ("has a mutation", spelled as state), - for a deletion
nucleotides	nuc.main.23063==T, nuc.HA.1000==G
coordinate systems	aa.GII.VP1.333==S — the reference, omitted when an organism has only one
insertions	ins.aa.S.214==EPE, ins.nuc.main.22204=match=GAGCC.*
regex	submittingGroup=match='^(ETH|Charité).*'
boolean	AND / OR / nested groups, with NOT either on groups or pushed onto the operators

…and twenty design decisions about how it is spelled and what it means. Each one is a radio group with the trade-off written next to it, and the tabs on the right show the consequences:

• URL — the query, the request, the encoding overhead, and whether it survives a round trip through the parser.
• Choice by choice — the same query rendered under every option of every syntax decision, so you can see which choices this query can even tell apart.
• Consequences — a linter over (query × dialect): things that cannot be expressed, traps, and trade-offs worth writing into the spec.
• Grammar — EBNF generated from the current choices, plus the operator table and the list of characters that can no longer appear unquoted.
• Parse — paste any filter string; the parser accepts every dialect on offer and reports where a choice made parsing harder than it looks.
• Demo data — a few dozen hand-written records per organism, and a table of how many of them each semantic choice would return.

Five organisms are configured, each one chosen to break a different shorthand:

organism	segments	references	what nuc.X.123 can mean
SARS-CoV-2	1	1	position 123 — nothing to disambiguate
Influenza A/H5N1	8	1	X is a segment
Influenza, segments 1…8	8	1	X is a segment, and nuc.4.501 reads as a decimal
Norovirus	1	2 (GI, GII)	X is a reference
CCHF	3	2	unresolvable by shape — the config has to be consulted

The last row is the point: the reference and the segment can each be made omissible safely, but not both at once. Amino-acid selectors never have this problem, because the gene is mandatory; it is the optional segment that makes the nucleotide form variable-length in two independent places.

Design decisions on the table

Syntax: AND/OR spelling · equality (= vs ==) · comparison operators (>= vs =ge=) · list literals · residue addressing · optional segment · reference in the selector · spelling of the loose comparisons · NOT · regex operator · quoting · quote character · whitespace · grouping parentheses · range spelling.

Semantics: case sensitivity · ambiguous residues · negation over unknowns · imprecise dates · how much regex the backend promises.

A few things that fall out of playing with it

Recorded here as observations, not recommendations:

• Residue conditions filter for state, never for a change relative to a reference. Gene and position are schema — the same for every record, enumerable from the annotation — while the residue is the datum, so the split matches how metadata already works and aa.S.501 behaves as an ordinary sparse column. Two things follow, and both are implemented here: there is no "any mutation" wildcard (aa.S.681!=P says it, and says which residue it treats as wildtype), and there is no reference base in the syntax (S:N501Y's N asserts something about the reference, not about the sequence being filtered). A query is then stable across reference updates. The cost lands on whoever writes the query: they need the reference to know the wildtype, which is why the builder shows wt P beside each position and gives you a ≠ wt button.

• Legacy queries can still be read, but only by someone holding the reference. Paste aminoAcidMutations=S:681 or nucleotideMutations=A23063T into the Parse tab: the first is rewritten to aa.S.681!=P and the second drops its reference base, both with an explanation. nucleotideMutations=C23063T is an error, because C contradicts the reference. That translation is a server-side job, and it is a good argument for doing the migration in one place rather than in every client.

• , for OR and , for lists is resolvable but not readable. A parser disambiguates by position (a ( immediately after an operator opens a list). Every human skimming a long query, and every client that splits on commas, does not.

• Unquoted values plus keyword connectives is the sharpest edge. country==United States cannot parse: the value ends at the space. Three of the parser fixtures in test/run.js are marked as expected failures for exactly this reason.

• Dropping "any mutation" makes the unknown-handling rule load-bearing. != <wildtype> is now the way to ask for a mutation, so what != does at a position that was never sequenced is no longer an edge case. Excluding unknowns is the right answer for that idiom — you have not observed a mutation there — but then == and != do not partition the dataset, and De Morgan rewriting, the thing that makes operator-level negation sufficient, silently changes the result set. Including them keeps NOT sound and reports every low-coverage genome as carrying every mutation it was never read for. The "Demo data" tab counts the difference; test/run.js asserts it.

• =range=(a,b) cannot say whether the endpoints are included, and the two-comparison form (d>=a;d<=b) gets it for free. Set an endpoint to exclusive in the builder and the linter will point out that the pair form drops the information.

• An imprecise date and an ambiguous residue are the same problem. In both cases the datum denotes a set of possibilities — every day in March, every base at an unread position — and a comparison has two honest readings: does this hold for the whole set, or for some member of it. Once that is the framing, one rule generates every date operator: "certainly" is set containment and "possibly" is set intersection. It also explains why == on a month has to mean containment rather than string equality, so sampleCollectionDate==2025-03 returns everything collected in March. And it is an argument for spelling the two the same way: =maybe= for a residue and =mayge= for a date, rather than a tilde for one and a word for the other.

• The loose reading breaks additivity. Under "possibly", a record known only to 2025-03 is returned by both halves of March, so counts over disjoint windows sum to more than the dataset. Under "certainly" it is returned by neither, so they sum to less. There is no reading that both keeps every record and partitions them; test/run.js asserts both halves of that.

• Adding a reference is cheap in the selector and expensive in the value. aa.GII.VP1.333==S grows one level; the LAPIS-style value form becomes insertions==GII:main:1000:AAT, four positional fields told apart by counting colons from the right. And a dot-separated selector quietly forbids reference names from containing dots — which rules out naming them after their accession (NC_045512.2), the way everyone names them.

• A per-condition reference lets a query name two coordinate systems at once. Whether aa.GII.VP1.333==S;aa.GI.VP1.333!=A is answerable depends on a data-model question the syntax never asks: is every sequence aligned to every reference, or only to the one it best matches? The query-scoped option (&reference=GII) makes the question impossible to ask, at the cost of a filter string that no longer means anything on its own.

Layout

index.html      structure and script tags
styles.css
js/schema.js    organisms, metadata fields, operators, which operators apply to which type
js/dialect.js   the sixteen design choices, their trade-offs, and four named presets
js/query.js     query tree, serialiser, De Morgan / flatten transforms
js/parse.js     tolerant parser: accepts every dialect, reports ambiguities
js/data.js      the demo dataset
js/evaluate.js  evaluates a query tree against it, honouring the semantic choices
js/lint.js      the consequences engine + round-trip check
js/grammar.js   EBNF generated from the current choices
js/examples.js  starting-point queries
js/ui.js        builder, choice panel, output tabs
test/           node smoke tests (no DOM) and a jsdom click-through

Adding a field, an organism, an operator or a design choice means editing one of schema.js, data.js or dialect.js; the rest of the app picks it up.