Skip to content

Create new test type aamtest for accessibility API testing #57696

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
spectranaut wants to merge 38 commits into
base: master
Choose a base branch
from

Fix tables and links

d4fca40
Select commit
Loading
Failed to load commit list.
Open

Create new test type aamtest for accessibility API testing #57696

Fix tables and links
d4fca40
Select commit
Loading
Failed to load commit list.
Community-TC Integration / infrastructure/ tests (firefox) failed Apr 28, 2026 in 21m 53s

Community-TC (pull_request)

Smoketests for wptrunner in firefox.

Details

View task in Taskcluster | View logs in Taskcluster | View task group in Taskcluster

Task Status

Started: 2026-04-28T20:22:00.834Z
Resolved: 2026-04-28T20:30:12.542Z
Task Execution Time: 8 minutes, 11 seconds, 708 milliseconds
Task Status: failed
Reason Resolved: failed
TaskId: dbqXHRnJTVmRJb5z3hz_bQ
RunId: 0

Artifacts

- public/logs/live_backing.log
- public/logs/live.log
- public/results/checkrun.md
- public/results/wptreport-firefox.json

WPT Command: python3 ./wpt run --log-tbpl=- --log-wptreport=../artifacts/wptreport-firefox.json --logcat-dir=../artifacts/ --metadata=infrastructure/metadata/ --include=infrastructure/ --log-mach-level=info --log-mach=- -y --no-pause --no-restart-on-unexpected --install-fonts --verify-log-full --no-headless --binary=/home/test/build/firefox/firefox firefox

[taskcluster 2026-04-28 20:22:00.909Z] Task ID: dbqXHRnJTVmRJb5z3hz_bQ
[taskcluster 2026-04-28 20:22:00.909Z] Worker ID: 1988358527258504351
[taskcluster 2026-04-28 20:22:00.909Z] Worker Group: us-east1-d
[taskcluster 2026-04-28 20:22:00.909Z] Worker Node Type: projects/757942385826/machineTypes/n2-standard-4
[taskcluster 2026-04-28 20:22:00.909Z] Worker Pool: proj-wpt/ci
[taskcluster 2026-04-28 20:22:00.909Z] Worker Version: 46.1.0
[taskcluster 2026-04-28 20:22:00.909Z] Public IP: 34.139.64.22
[taskcluster 2026-04-28 20:22:00.909Z] Hostname: proj-wpt-ci-ezvhiq0isdsyktktuqr-qw

b1e71f63927a - Started downloading
b1e71f63927a - Downloaded in 0.034 seconds
66587c81b81a - Started downloading
5b82b858ea7e - Started downloading
08c7e224535e - Started downloading
08c7e224535e - Downloaded in 0.042 seconds
66587c81b81a - Downloaded in 0.207 seconds
4f4fb700ef54 - Started downloading
4f4fb700ef54 - Downloaded in 0.002 seconds
32bbaaf98e4c - Started downloading
dd5257950bd4 - Started downloading

...(4332 lines hidden)...

PID 19883 | JavaScript error: http://web-platform.test:8000/infrastructure/test262/module-failure.test262-module-test.html, line 18: uncaught exception: Test262Error: a should be 2 in module Expected SameValue(«1», «2») to be true
TEST-FAIL | /infrastructure/test262/module-failure.test262-module.html | Test - a should be 2 in module Expected SameValue(«1», «2») to be true
window.test262HarnessDone<@http://web-platform.test:8000/infrastructure/test262/module-failure.test262-module.html:10:15
Test.prototype.step@http://web-platform.test:8000/resources/testharness.js:2869:25
Test.prototype.step_func_done/<@http://web-platform.test:8000/resources/testharness.js:2944:32
done@http://web-platform.test:8000/resources/test262/test262-reporter.js:99:22
on_error@http://web-platform.test:8000/resources/test262/test262-reporter.js:156:9
 5:35.17 TEST_END: Test OK. Subtests passed 0/1. Unexpected 0
TEST-OK | /infrastructure/test262/module-failure.test262-module.html | took 224ms
 5:35.17 TEST_START: /infrastructure/test262/module.test262-module.html
TEST-START | /infrastructure/test262/module.test262-module.html
 5:35.17 INFO Closing window d10d8a6a-6ae9-45a2-8f2d-f29c3e143f66
Closing window d10d8a6a-6ae9-45a2-8f2d-f29c3e143f66
TEST-PASS | /infrastructure/test262/module.test262-module.html | Test
 5:35.40 TEST_END: Test OK. Subtests passed 1/1. Unexpected 0
TEST-OK | /infrastructure/test262/module.test262-module.html | took 235ms
 5:35.40 TEST_START: /infrastructure/test262/negative-early.test262.html
TEST-START | /infrastructure/test262/negative-early.test262.html
 5:35.41 INFO Closing window dd38ce68-2b41-40d8-a581-38757e6df8e8
Closing window dd38ce68-2b41-40d8-a581-38757e6df8e8
 5:35.59 pid:19883 JavaScript error: http://web-platform.test:8000/infrastructure/test262/negative-early.js, line 9: SyntaxError: let is a reserved identifier
PID 19883 | JavaScript error: http://web-platform.test:8000/infrastructure/test262/negative-early.js, line 9: SyntaxError: let is a reserved identifier
TEST-PASS | /infrastructure/test262/negative-early.test262.html | Test
 5:35.61 TEST_END: Test OK. Subtests passed 1/1. Unexpected 0
TEST-OK | /infrastructure/test262/negative-early.test262.html | took 208ms
 5:35.61 TEST_START: /infrastructure/test262/negative-error.test262.html
TEST-START | /infrastructure/test262/negative-error.test262.html
 5:35.62 INFO Closing window 49666b58-d7c9-4af9-b588-271f2fad66e3
Closing window 49666b58-d7c9-4af9-b588-271f2fad66e3
 5:35.81 pid:19883 JavaScript error: http://web-platform.test:8000/infrastructure/test262/negative-error.js, line 8: RangeError:
PID 19883 | JavaScript error: http://web-platform.test:8000/infrastructure/test262/negative-error.js, line 8: RangeError:
TEST-FAIL | /infrastructure/test262/negative-error.test262.html | Test - Expected TypeError but got RangeError: 
window.test262HarnessDone<@http://web-platform.test:8000/infrastructure/test262/negative-error.test262.html:12:15
Test.prototype.step@http://web-platform.test:8000/resources/testharness.js:2869:25
Test.prototype.step_func_done/<@http://web-platform.test:8000/resources/testharness.js:2944:32
done@http://web-platform.test:8000/resources/test262/test262-reporter.js:99:22
on_error@http://web-platform.test:8000/resources/test262/test262-reporter.js:156:9
 5:35.84 TEST_END: Test OK. Subtests passed 0/1. Unexpected 0
TEST-OK | /infrastructure/test262/negative-error.test262.html | took 226ms
 5:35.84 TEST_START: /infrastructure/test262/negative-failure.test262.html
TEST-START | /infrastructure/test262/negative-failure.test262.html
 5:35.85 INFO Closing window 0bbaf9c2-0c50-4e61-b5e8-994794c41f09
Closing window 0bbaf9c2-0c50-4e61-b5e8-994794c41f09
TEST-FAIL | /infrastructure/test262/negative-failure.test262.html | Test - Expected Test262Error but test completed without error.
window.test262HarnessDone<@http://web-platform.test:8000/infrastructure/test262/negative-failure.test262.html:10:15
Test.prototype.step@http://web-platform.test:8000/resources/testharness.js:2869:25
Test.prototype.step_func_done/<@http://web-platform.test:8000/resources/testharness.js:2944:32
done@http://web-platform.test:8000/resources/test262/test262-reporter.js:99:22
@http://web-platform.test:8000/resources/test262/test262-reporter.js:105:13
 5:36.06 TEST_END: Test OK. Subtests passed 0/1. Unexpected 0
TEST-OK | /infrastructure/test262/negative-failure.test262.html | took 218ms
 5:36.06 TEST_START: /infrastructure/test262/negative-parse.test262.html
TEST-START | /infrastructure/test262/negative-parse.test262.html
 5:36.07 INFO Closing window d25bfdfb-de41-4bb7-86c8-4c655cb4e87e
Closing window d25bfdfb-de41-4bb7-86c8-4c655cb4e87e
 5:36.26 pid:19883 JavaScript error: http://web-platform.test:8000/infrastructure/test262/negative-parse.js, line 8: SyntaxError: illegal character U+0040
PID 19883 | JavaScript error: http://web-platform.test:8000/infrastructure/test262/negative-parse.js, line 8: SyntaxError: illegal character U+0040
TEST-PASS | /infrastructure/test262/negative-parse.test262.html | Test
 5:36.28 TEST_END: Test OK. Subtests passed 1/1. Unexpected 0
TEST-OK | /infrastructure/test262/negative-parse.test262.html | took 222ms
 5:36.28 TEST_START: /infrastructure/test262/negative-resolution.test262-module.html
TEST-START | /infrastructure/test262/negative-resolution.test262-module.html
 5:36.29 INFO Closing window c20605e3-694b-4e18-a063-276c3e1db5bb
Closing window c20605e3-694b-4e18-a063-276c3e1db5bb
 5:36.48 pid:19883 JavaScript error: http://web-platform.test:8000/infrastructure/test262/support/invalid-module.js, line 1: SyntaxError: illegal character U+0040
PID 19883 | JavaScript error: http://web-platform.test:8000/infrastructure/test262/support/invalid-module.js, line 1: SyntaxError: illegal character U+0040
TEST-PASS | /infrastructure/test262/negative-resolution.test262-module.html | Test
 5:36.51 TEST_END: Test OK. Subtests passed 1/1. Unexpected 0
TEST-OK | /infrastructure/test262/negative-resolution.test262-module.html | took 234ms
 5:36.51 TEST_START: /infrastructure/test262/negative.test262.html
TEST-START | /infrastructure/test262/negative.test262.html
 5:36.52 INFO Closing window f298483c-93f1-4b93-bfc4-8b5f642b0974
Closing window f298483c-93f1-4b93-bfc4-8b5f642b0974
 5:36.71 pid:19883 JavaScript error: http://web-platform.test:8000/infrastructure/test262/negative.js, line 7: uncaught exception: Test262Error:
PID 19883 | JavaScript error: http://web-platform.test:8000/infrastructure/test262/negative.js, line 7: uncaught exception: Test262Error:
TEST-PASS | /infrastructure/test262/negative.test262.html | Test
 5:36.73 TEST_END: Test OK. Subtests passed 1/1. Unexpected 0
TEST-OK | /infrastructure/test262/negative.test262.html | took 214ms
 5:36.73 TEST_START: /infrastructure/test262/strict-error.test262.strict.html
TEST-START | /infrastructure/test262/strict-error.test262.strict.html
 5:36.73 INFO Closing window 0fe23eb0-d937-4d42-b275-b302b4fd69ac
Closing window 0fe23eb0-d937-4d42-b275-b302b4fd69ac
 5:36.93 pid:19883 JavaScript error: http://web-platform.test:8000/infrastructure/test262/strict-error.test262.strict.js, line 8: ReferenceError: foo is not defined
PID 19883 | JavaScript error: http://web-platform.test:8000/infrastructure/test262/strict-error.test262.strict.js, line 8: ReferenceError: foo is not defined
TEST-FAIL | /infrastructure/test262/strict-error.test262.strict.html | Test - ReferenceError: foo is not defined
window.test262HarnessDone<@http://web-platform.test:8000/infrastructure/test262/strict-error.test262.strict.html:12:15
Test.prototype.step@http://web-platform.test:8000/resources/testharness.js:2869:25
Test.prototype.step_func_done/<@http://web-platform.test:8000/resources/testharness.js:2944:32
done@http://web-platform.test:8000/resources/test262/test262-reporter.js:99:22
on_error@http://web-platform.test:8000/resources/test262/test262-reporter.js:156:9
 5:36.95 TEST_END: Test OK. Subtests passed 0/1. Unexpected 0
TEST-OK | /infrastructure/test262/strict-error.test262.strict.html | took 218ms
 5:36.95 TEST_START: /infrastructure/test262/strict-failure.test262.strict.html
TEST-START | /infrastructure/test262/strict-failure.test262.strict.html
 5:36.95 INFO Closing window 134a60cd-ee12-4a30-b5a8-177688193d6b
Closing window 134a60cd-ee12-4a30-b5a8-177688193d6b
 5:37.15 pid:19883 JavaScript error: http://web-platform.test:8000/third_party/test262/harness/assert.js, line 49: uncaught exception: Test262Error: One should be two in strict mode Expected SameValue(«1», «2») to be true
PID 19883 | JavaScript error: http://web-platform.test:8000/third_party/test262/harness/assert.js, line 49: uncaught exception: Test262Error: One should be two in strict mode Expected SameValue(«1», «2») to be true
TEST-FAIL | /infrastructure/test262/strict-failure.test262.strict.html | Test - One should be two in strict mode Expected SameValue(«1», «2») to be true
window.test262HarnessDone<@http://web-platform.test:8000/infrastructure/test262/strict-failure.test262.strict.html:10:15
Test.prototype.step@http://web-platform.test:8000/resources/testharness.js:2869:25
Test.prototype.step_func_done/<@http://web-platform.test:8000/resources/testharness.js:2944:32
done@http://web-platform.test:8000/resources/test262/test262-reporter.js:99:22
on_error@http://web-platform.test:8000/resources/test262/test262-reporter.js:156:9
 5:37.18 TEST_END: Test OK. Subtests passed 0/1. Unexpected 0
TEST-OK | /infrastructure/test262/strict-failure.test262.strict.html | took 227ms
 5:37.18 TEST_START: /infrastructure/test262/strict.test262.html
TEST-START | /infrastructure/test262/strict.test262.html
 5:37.18 INFO Closing window be58f9b4-9917-439d-80b7-308c22b860b6
Closing window be58f9b4-9917-439d-80b7-308c22b860b6
 5:37.38 pid:19883 JavaScript error: http://web-platform.test:8000/infrastructure/test262/strict.js, line 8: SyntaxError: invalid escape sequence
PID 19883 | JavaScript error: http://web-platform.test:8000/infrastructure/test262/strict.js, line 8: SyntaxError: invalid escape sequence
TEST-PASS | /infrastructure/test262/strict.test262.html | Test
 5:37.40 TEST_END: Test OK. Subtests passed 1/1. Unexpected 0
TEST-OK | /infrastructure/test262/strict.test262.html | took 221ms
 5:37.40 TEST_START: /infrastructure/test262/timeout.test262.html
TEST-START | /infrastructure/test262/timeout.test262.html
 5:37.40 INFO Closing window 6c594031-c3ca-499d-9331-1e3a9040b83c
Closing window 6c594031-c3ca-499d-9331-1e3a9040b83c
mem avail: 13835 of 15806 MiB (87.53%), swap free:    0 of    0 MiB ( 0.00%)
TEST-NOTRUN | /infrastructure/test262/timeout.test262.html | Test
 5:47.51 TEST_END: Test TIMEOUT. Subtests passed 0/1. Unexpected 0
TEST-TIMEOUT | /infrastructure/test262/timeout.test262.html | took 10115ms
 5:47.51 TEST_START: /infrastructure/test262/unexpected-error.test262.html
TEST-START | /infrastructure/test262/unexpected-error.test262.html
 5:47.52 INFO Closing window e768df8d-c388-40c8-94a1-4e14ef2ac998
Closing window e768df8d-c388-40c8-94a1-4e14ef2ac998
 5:47.70 pid:19883 JavaScript error: http://web-platform.test:8000/infrastructure/test262/unexpected-error.js, line 5: ReferenceError: foo is not defined
PID 19883 | JavaScript error: http://web-platform.test:8000/infrastructure/test262/unexpected-error.js, line 5: ReferenceError: foo is not defined
TEST-FAIL | /infrastructure/test262/unexpected-error.test262.html | Test - ReferenceError: foo is not defined
window.test262HarnessDone<@http://web-platform.test:8000/infrastructure/test262/unexpected-error.test262.html:12:15
Test.prototype.step@http://web-platform.test:8000/resources/testharness.js:2869:25
Test.prototype.step_func_done/<@http://web-platform.test:8000/resources/testharness.js:2944:32
done@http://web-platform.test:8000/resources/test262/test262-reporter.js:99:22
on_error@http://web-platform.test:8000/resources/test262/test262-reporter.js:156:9
 5:47.72 TEST_END: Test OK. Subtests passed 0/1. Unexpected 0
TEST-OK | /infrastructure/test262/unexpected-error.test262.html | took 206ms
 5:47.72 INFO No more tests
No more tests
 5:47.72 pid:19883 1777408208106	Marionette	INFO	Stopped listening on port 48749
PID 19883 | 1777408208106	Marionette	INFO	Stopped listening on port 48749
 5:48.18 INFO Browser exited with return code 0
Browser exited with return code 0
 5:48.01 INFO Closing logging queue
Closing logging queue
 5:48.01 INFO queue closed
queue closed
 5:48.20 SUITE_END

web-platform-test
~~~~~~~~~~~~~~~~~
Ran 453 checks (244 subtests, 209 tests)
Expected results: 447
Skipped: 2 tests
Unexpected results: 4
  subtest: 4 (4 fail)

Error Summary
-------------
/infrastructure/testdriver/virtual_authenticator.html
  FAIL Can add a credential - promise_test: Unhandled rejection with value: "error: Action add_credential failed: TypeError: credentials is undefined\nstacktrace:\n\twebAuthn_addCredential@chrome://remote/content/marionette/driver.sys.mjs:3782:9\n\tdispatch@chrome://remote/content/marionette/server.sys.mjs:317:49\n\texecute@chrome://remote/content/marionette/server.sys.mjs:289:16\n\tonPacket/<@chrome://remote/content/marionette/server.sys.mjs:262:20\n\tonPacket@chrome://remote/content/marionette/server.sys.mjs:263:9\n\t_onJSONObjectReady/<@chrome://remote/content/marionette/transport.sys.mjs:494:20\n"
  FAIL Can get the credentials - assert_equals: expected 1 but got 0
@http://web-platform.test:8000/infrastructure/testdriver/virtual_authenticator.html:49:16
  FAIL Can remove a credential - promise_test: Unhandled rejection with value: "error: Action remove_credential failed: [Exception... \"Component returned failure code: 0x80070057 (NS_ERROR_ILLEGAL_VALUE) [nsIWebAuthnService.removeCredential]\"  nsresult: \"0x80070057 (NS_ERROR_ILLEGAL_VALUE)\"  location: \"JS frame :: chrome://remote/content/marionette/webauthn.sys.mjs :: webauthn.removeCredential :: line 158\"  data: no]\nstacktrace:\n\twebauthn.removeCredential@chrome://remote/content/marionette/webauthn.sys.mjs:158:24\n\twebAuthn_removeCredential@chrome://remote/content/marionette/driver.sys.mjs:3960:19\n\tdispatch@chrome://remote/content/marionette/server.sys.mjs:317:49\n\texecute@chrome://remote/content/marionette/server.sys.mjs:289:16\n\tonPacket/<@chrome://remote/content/marionette/server.sys.mjs:262:20\n\tonPacket@chrome://remote/content/marionette/server.sys.mjs:263:9\n\t_onJSONObjectReady/<@chrome://remote/content/marionette/transport.sys.mjs:494:20\n"
  FAIL Can remove all credentials - promise_test: Unhandled rejection with value: "error: Action add_credential failed: TypeError: credentials is undefined\nstacktrace:\n\twebAuthn_addCredential@chrome://remote/content/marionette/driver.sys.mjs:3782:9\n\tdispatch@chrome://remote/content/marionette/server.sys.mjs:317:49\n\texecute@chrome://remote/content/marionette/server.sys.mjs:289:16\n\tonPacket/<@chrome://remote/content/marionette/server.sys.mjs:262:20\n\tonPacket@chrome://remote/content/marionette/server.sys.mjs:263:9\n\t_onJSONObjectReady/<@chrome://remote/content/marionette/transport.sys.mjs:494:20\n"
SUITE-END | took 276s
 5:48.20 INFO Got 1 unexpected results, with 0 unexpected passes
Got 1 unexpected results, with 0 unexpected passes
 5:48.20 wptserve INFO Stopped http server on 127.0.0.1:8000
wptserve Stopped http server on 127.0.0.1:8000
 5:48.20 wptserve INFO Stopped http server on 127.0.0.1:8001
wptserve Stopped http server on 127.0.0.1:8001
 5:48.21 wptserve INFO Stopped http server on 127.0.0.1:8002
wptserve Stopped http server on 127.0.0.1:8002
 5:48.21 wptserve INFO Stopped http server on 127.0.0.1:8003
wptserve Stopped http server on 127.0.0.1:8003
 5:48.21 wptserve INFO Stopped http server on 127.0.0.1:8443
wptserve Stopped http server on 127.0.0.1:8443
 5:48.21 wptserve INFO Stopped http server on 127.0.0.1:8444
wptserve Stopped http server on 127.0.0.1:8444
 5:48.21 wptserve INFO Stopped http server on 127.0.0.1:8445
wptserve Stopped http server on 127.0.0.1:8445
 5:48.22 wptserve INFO Stopped http server on 127.0.0.1:8446
wptserve Stopped http server on 127.0.0.1:8446
 5:48.22 wptserve INFO Stopped http server on 127.0.0.1:9000
wptserve Stopped http server on 127.0.0.1:9000
 5:48.23 wptserve INFO Stopped WebTransport over HTTP/3 server on 127.0.0.1:11000
wptserve Stopped WebTransport over HTTP/3 server on 127.0.0.1:11000
 5:48.29 wptserve INFO Close on: (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('127.0.0.1', 8889))
wptserve Close on: (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('127.0.0.1', 8889))
 5:48.55 wptserve INFO Close on: (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('127.0.0.1', 8888))
wptserve Close on: (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('127.0.0.1', 8888))
 5:48.59 INFO Removed font: Ahem.ttf
Removed font: Ahem.ttf
 5:48.62 INFO Closing logging queue
Closing logging queue
 5:48.62 INFO queue closed
queue closed
[taskcluster 2026-04-28 20:30:10.834Z] === Task Finished ===
[taskcluster 2026-04-28 20:30:11.661Z] Unsuccessful task run with exit code: 1 completed in 490.752 seconds
View more details on Community-TC Integration