test: bump JSON-Schema-Test-Suite to current + auto-discovering harness - #10
Merged
Conversation
The test-suite submodule was pinned at a 2021 commit (754 commits behind);
bumping it broke every per-draft CT suite because each hard-coded one testcase
function per keyword file and matched skip entries by description.
- Bump test/JSON-Schema-Test-Suite to 0c7b65d (current upstream).
- Rework jesse_tests_util + all 5 draft suites: auto-discover *.json test
files by globbing and run them as a single aggregate 'conformance' case that
collects all failures (rather than aborting on the first) and prints a
pass/skip/fail summary. Adding/removing upstream keyword files now needs no
suite change. The harness also warns about stale skip_list entries whose
description no longer matches any case.
- Enumerate unsupported cases per draft in skip_list/0 with grouped reasons
(in-document $id base-URI resolution, $recursiveRef/$dynamicRef,
$vocabulary, remote-metaschema fetch, URN base URIs, ECMA \p{...} regex
property escapes).
- Fix maxProperties/minProperties to accept an integer-valued float (2.0) per
draft-06+, in draft6/2019-09/2020-12 validators.
- Repoint jesse's own remoteRefExtra fixtures at the relocated remote schema.
Compliance vs the current suite: draft3 446/447, draft4 664/671, draft6
827/839, 2019-09 1200/1234, 2020-12 1226/1276; all non-skipped assertions pass.
start_remotes_server/1 previously turned every unexpected inets:start(httpd,...) error into ok and accepted eaddrinuse without checking what was listening, so a harness/port failure could masquerade as a JSON Schema conformance failure or a suite could use a foreign process's document root. Now: an unexpected start error aborts init_per_suite; when the port is already bound (a previous suite in the same node), verify it by fetching a known fixture over HTTP and byte-comparing it to the file on disk, erroring loudly if the content differs or the server is unreachable.
Review follow-ups: - Make httpd ownership explicit: start_remotes_server/1 returns the service pid, which init_per_suite keeps in Config and end_per_suite passes to stop_remotes_server/1. Suites run sequentially so each owns port 1234 for its lifetime; any start error (incl. a leaked/foreign listener) aborts init_per_suite instead of being mistaken for a conformance failure. (The port is fixed at 1234 because the upstream refRemote schemas embed it literally, so a dynamic port would require rewriting every embedded URL.) - Run per-case setup (option parsing) inside the try/catch: an unknown/bad option now surfaces as that case's collected failure (with file/case info) rather than aborting the whole aggregate testcase via list_to_existing_atom.
Rework the Unreleased section into a concise 1.9.0 release entry: headline
draft 2019-09/2020-12 support, the draft-06 examples and maxProperties/
minProperties fixes, and the test-suite refresh. Drop the dev-facing compliance
tables and gap enumeration. Version is git-tag driven ({vsn, git}), so no
app.src change; the 1.9.0 tag is applied to the merge commit at release time.
thalesmg
reviewed
Jul 24, 2026
Comment on lines
-138
to
-154
| %% The original bug originated from starting from a map schema input, so it was | ||
| %% not triggered by `do_test', which loads the schema as proplists rather than | ||
| %% maps. | ||
| extends_smoke_test(_Config) -> | ||
| Schema = #{ | ||
| <<"$schema">> => <<"http://json-schema.org/draft-03/schema#">>, | ||
| <<"description">> => <<"a description">>, | ||
| <<"extends">> => | ||
| #{<<"properties">> => | ||
| #{<<"disallow">> => | ||
| #{<<"disallow">> => [<<"number">>], | ||
| <<"required">> => true}}}, | ||
| <<"id">> => <<"http://json-schema.org/draft-03/schema#">>, | ||
| <<"title">> => <<"title">>, | ||
| <<"type">> => <<"object">>}, | ||
| Data = #{<<"disallow">> => <<"a">>}, | ||
| ?assertEqual({ok, Data}, jesse:validate_with_schema(Schema, Data)). |
There was a problem hiding this comment.
this shouldn't be removed. it's not a standard conformance test. it was added to our fork and ignored by upstream.
Comment on lines
-191
to
-203
| %% see https://github.com/emqx/emqx/issues/17977 | ||
| examples_are_ignored(_Config) -> | ||
| Schema = #{ | ||
| <<"properties">> => | ||
| #{<<"name">> => | ||
| #{<<"examples">> => [<<"foo">>], | ||
| <<"type">> => <<"string">>}}, | ||
| <<"type">> => <<"object">>}, | ||
| Data = #{<<"name">> => <<"bar">>}, | ||
| ?assertMatch( | ||
| {ok, _}, | ||
| jesse:validate_with_schema(Schema, Data, []) | ||
| ). |
There was a problem hiding this comment.
please don't delete this. just fixed it recently. 😅
The auto-discovering conformance case only runs JSON-suite files, so two
hand-written .erl testcases were inadvertently dropped when the suites were
rewritten:
- draft6 examples_are_ignored (regression for the draft-06 examples fix,
emqx/emqx#17977)
- draft3 extends_smoke_test
Restore both alongside the conformance case.
thalesmg
approved these changes
Jul 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Updates the JSON-Schema-Test-Suite submodule from a 2021 commit (754
commits / 5 years behind) to current upstream, and reworks the CT harness so
this and future bumps don't break the suites. Addresses the staleness flagged
in #9. jesse now tracks the spec far more closely.
Why a bump used to break everything
Each
_SUITE.erlhard-coded one testcase function per keyword file and matchedskip entries by case description. Over 754 upstream commits, files were
renamed/removed and descriptions drifted, so a naive bump produced ~86 failures
across all 5 suites (
badmatchon gone files, stale skips, plus genuinely newcases).
Harness rework (future-proof)
jesse_tests_util+ all 5 draft suites now auto-discover*.jsonfilesby globbing and run them as a single aggregate
conformancecase thatcollects every failure (instead of aborting on the first) and prints a
total / passed / skipped / failedsummary with each failure's file + case.skip_listentries whose description nolonger matches any case, so drift is visible on the next bump.
skip_list/0, grouped withreasons — never silently dropped.
Real conformance fixes
maxProperties/minPropertieswith an integer-valued float (e.g.2.0):draft-06+ treats a zero-fraction number as an integer; jesse's
is_integerguard wrongly rejected it as an invalid schema. Fixed in draft6/2019-09/2020-12.
remoteRefExtraregression fixtures at the relocatedupstream remote schema.
Compliance against the current suite
All non-skipped assertions pass. The skipped remainder are documented gaps, by
category:
$idbase-URI resolution (the large one: location-independentidentifiers, nearest-parent
$idscoping,$ref/anchor resolved against$idbases,ref to if/then/else, remote base-URI changes) — needs a real$id→subschema resolver; candidate for a dedicated follow-up.$recursiveRef/$dynamicRef(dynamic scope stack) — hard-error today.$vocabulary/ custom metaschemas; remote-metaschema fetch.urn:base URIs; empty-string reference tokens.\p{...}unicode property escapes — Erlang'sre/PCRE only acceptsshort names (
\p{L}), so\p{Letter}won't compile.Validation
make ct— all 5 draft suites green (aggregate conformance case each)make eunit— cleanmake xref— cleanFollow-ups
$idbase-URI resolution — the biggest remaining compliancegap (~50 cases across drafts); warrants its own PR.
patternregex currently crashes jesse'sre_run(badarg) rather than erroring cleanly; worth a defensive guard,plus optional ECMA→PCRE property-name translation for
\p{...}.