Skip to content

Create new test type aamtest for accessibility API testing #57696

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
spectranaut wants to merge 38 commits into
base: master
Choose a base branch
from

Fix tables and links

d4fca40
Select commit
Loading
Failed to load commit list.
Open

Create new test type aamtest for accessibility API testing #57696

Fix tables and links
d4fca40
Select commit
Loading
Failed to load commit list.
Community-TC Integration / infrastructure/ tests (firefox_android) failed Apr 28, 2026 in 33m 59s

Community-TC (pull_request)

Smoketests for wptrunner in firefox_android.

Details

View task in Taskcluster | View logs in Taskcluster | View task group in Taskcluster

Task Status

Started: 2026-04-28T20:21:16.610Z
Resolved: 2026-04-28T20:42:19.432Z
Task Execution Time: 21 minutes, 2 seconds, 822 milliseconds
Task Status: failed
Reason Resolved: failed
TaskId: fpp7QybRStaEP61tg4QBPA
RunId: 0

Artifacts

- public/logs/live_backing.log
- public/logs/live.log
- public/results/checkrun.md
- public/results/emulator.log
- public/results/logcat-emulator-5554.log
- public/results/wptreport-firefox_android.json

WPT Command: python3 ./wpt run --log-tbpl=- --log-wptreport=../artifacts/wptreport-firefox_android.json --logcat-dir=../artifacts/ --metadata=infrastructure/metadata/ --include=infrastructure/ --log-mach-level=info --log-mach=- -y --no-pause --no-restart-on-unexpected --install-fonts --verify-log-full --no-headless --install-browser --install-webdriver --logcat-dir /home/test/artifacts/ firefox_android

[taskcluster 2026-04-28 20:21:16.714Z] Task ID: fpp7QybRStaEP61tg4QBPA
[taskcluster 2026-04-28 20:21:16.714Z] Worker ID: 1711306069782607787
[taskcluster 2026-04-28 20:21:16.714Z] Worker Group: us-east4-b
[taskcluster 2026-04-28 20:21:16.714Z] Worker Node Type: projects/757942385826/machineTypes/n2-standard-4
[taskcluster 2026-04-28 20:21:16.714Z] Worker Pool: proj-wpt/ci
[taskcluster 2026-04-28 20:21:16.714Z] Worker Version: 46.1.0
[taskcluster 2026-04-28 20:21:16.714Z] Public IP: 35.188.224.238
[taskcluster 2026-04-28 20:21:16.714Z] Hostname: proj-wpt-ci-ws3t4qd2txeqthywmt4q9a

[taskcluster 2026-04-28 20:21:17.809Z] === Task Starting ===
+ '[' '' ']'
+ '[' -d /etc/profile.d ']'
+ for i in /etc/profile.d/*.sh
+ '[' -r /etc/profile.d/01-locale-fix.sh ']'
+ . /etc/profile.d/01-locale-fix.sh
+++ /usr/bin/locale-check C.UTF-8
++ eval
+ for i in /etc/profile.d/*.sh
+ '[' -r /etc/profile.d/apps-bin-path.sh ']'
+ . /etc/profile.d/apps-bin-path.sh

...(4589 lines hidden)...

16:51.23 TEST_START: /infrastructure/test262/module-error.test262-module.html
TEST-START | /infrastructure/test262/module-error.test262-module.html
16:51.24 INFO Closing window a816b6c0-3566-4f2b-b339-34689b8d9626
Closing window a816b6c0-3566-4f2b-b339-34689b8d9626
TEST-FAIL | /infrastructure/test262/module-error.test262-module.html | Test - ReferenceError: foo is not defined
window.test262HarnessDone<@http://web-platform.test:8000/infrastructure/test262/module-error.test262-module.html:12:15
Test.prototype.step@http://web-platform.test:8000/resources/testharness.js:2869:25
Test.prototype.step_func_done/<@http://web-platform.test:8000/resources/testharness.js:2944:32
done@http://web-platform.test:8000/resources/test262/test262-reporter.js:99:22
on_error@http://web-platform.test:8000/resources/test262/test262-reporter.js:156:9
16:51.65 TEST_END: Test OK. Subtests passed 0/1. Unexpected 0
TEST-OK | /infrastructure/test262/module-error.test262-module.html | took 420ms
16:51.65 TEST_START: /infrastructure/test262/module-failure.test262-module.html
TEST-START | /infrastructure/test262/module-failure.test262-module.html
16:51.66 INFO Closing window eadf0d9f-2cba-4215-a51a-bfe27d664394
Closing window eadf0d9f-2cba-4215-a51a-bfe27d664394
TEST-FAIL | /infrastructure/test262/module-failure.test262-module.html | Test - a should be 2 in module Expected SameValue(«1», «2») to be true
window.test262HarnessDone<@http://web-platform.test:8000/infrastructure/test262/module-failure.test262-module.html:10:15
Test.prototype.step@http://web-platform.test:8000/resources/testharness.js:2869:25
Test.prototype.step_func_done/<@http://web-platform.test:8000/resources/testharness.js:2944:32
done@http://web-platform.test:8000/resources/test262/test262-reporter.js:99:22
on_error@http://web-platform.test:8000/resources/test262/test262-reporter.js:156:9
16:52.09 TEST_END: Test OK. Subtests passed 0/1. Unexpected 0
TEST-OK | /infrastructure/test262/module-failure.test262-module.html | took 439ms
16:52.09 TEST_START: /infrastructure/test262/module.test262-module.html
TEST-START | /infrastructure/test262/module.test262-module.html
16:52.10 INFO Closing window 01e3464b-c4f9-4160-9418-641e3fcca332
Closing window 01e3464b-c4f9-4160-9418-641e3fcca332
TEST-PASS | /infrastructure/test262/module.test262-module.html | Test
16:52.56 TEST_END: Test OK. Subtests passed 1/1. Unexpected 0
TEST-OK | /infrastructure/test262/module.test262-module.html | took 466ms
16:52.56 TEST_START: /infrastructure/test262/negative-early.test262.html
TEST-START | /infrastructure/test262/negative-early.test262.html
16:52.56 INFO Closing window fbcce9b9-9db1-47ec-85e8-4e8e5a4710b4
Closing window fbcce9b9-9db1-47ec-85e8-4e8e5a4710b4
TEST-PASS | /infrastructure/test262/negative-early.test262.html | Test
16:53.00 TEST_END: Test OK. Subtests passed 1/1. Unexpected 0
TEST-OK | /infrastructure/test262/negative-early.test262.html | took 443ms
16:53.00 TEST_START: /infrastructure/test262/negative-error.test262.html
TEST-START | /infrastructure/test262/negative-error.test262.html
16:53.01 INFO Closing window b7cb32ee-7e15-4c19-a6bc-eb5426c7fb35
Closing window b7cb32ee-7e15-4c19-a6bc-eb5426c7fb35
TEST-FAIL | /infrastructure/test262/negative-error.test262.html | Test - Expected TypeError but got RangeError: 
window.test262HarnessDone<@http://web-platform.test:8000/infrastructure/test262/negative-error.test262.html:12:15
Test.prototype.step@http://web-platform.test:8000/resources/testharness.js:2869:25
Test.prototype.step_func_done/<@http://web-platform.test:8000/resources/testharness.js:2944:32
done@http://web-platform.test:8000/resources/test262/test262-reporter.js:99:22
on_error@http://web-platform.test:8000/resources/test262/test262-reporter.js:156:9
16:53.39 TEST_END: Test OK. Subtests passed 0/1. Unexpected 0
TEST-OK | /infrastructure/test262/negative-error.test262.html | took 393ms
16:53.39 TEST_START: /infrastructure/test262/negative-failure.test262.html
TEST-START | /infrastructure/test262/negative-failure.test262.html
16:53.40 INFO Closing window 88406080-d984-424e-844d-1323b17afb7a
Closing window 88406080-d984-424e-844d-1323b17afb7a
TEST-FAIL | /infrastructure/test262/negative-failure.test262.html | Test - Expected Test262Error but test completed without error.
window.test262HarnessDone<@http://web-platform.test:8000/infrastructure/test262/negative-failure.test262.html:10:15
Test.prototype.step@http://web-platform.test:8000/resources/testharness.js:2869:25
Test.prototype.step_func_done/<@http://web-platform.test:8000/resources/testharness.js:2944:32
done@http://web-platform.test:8000/resources/test262/test262-reporter.js:99:22
@http://web-platform.test:8000/resources/test262/test262-reporter.js:105:13
16:53.83 TEST_END: Test OK. Subtests passed 0/1. Unexpected 0
TEST-OK | /infrastructure/test262/negative-failure.test262.html | took 439ms
16:53.83 TEST_START: /infrastructure/test262/negative-parse.test262.html
TEST-START | /infrastructure/test262/negative-parse.test262.html
16:53.84 INFO Closing window 6c5fcf64-6de2-497a-a2ac-d6396f8e81a4
Closing window 6c5fcf64-6de2-497a-a2ac-d6396f8e81a4
TEST-PASS | /infrastructure/test262/negative-parse.test262.html | Test
16:54.27 TEST_END: Test OK. Subtests passed 1/1. Unexpected 0
TEST-OK | /infrastructure/test262/negative-parse.test262.html | took 436ms
16:54.27 TEST_START: /infrastructure/test262/negative-resolution.test262-module.html
TEST-START | /infrastructure/test262/negative-resolution.test262-module.html
16:54.27 INFO Closing window 78ac8651-402a-4f12-8954-c30cd5252678
Closing window 78ac8651-402a-4f12-8954-c30cd5252678
TEST-PASS | /infrastructure/test262/negative-resolution.test262-module.html | Test
16:54.72 TEST_END: Test OK. Subtests passed 1/1. Unexpected 0
TEST-OK | /infrastructure/test262/negative-resolution.test262-module.html | took 450ms
16:54.72 TEST_START: /infrastructure/test262/negative.test262.html
TEST-START | /infrastructure/test262/negative.test262.html
16:54.73 INFO Closing window 288c928d-81de-4e34-88b9-dacbf327b833
Closing window 288c928d-81de-4e34-88b9-dacbf327b833
TEST-PASS | /infrastructure/test262/negative.test262.html | Test
16:55.21 TEST_END: Test OK. Subtests passed 1/1. Unexpected 0
TEST-OK | /infrastructure/test262/negative.test262.html | took 489ms
16:55.21 TEST_START: /infrastructure/test262/strict-error.test262.strict.html
TEST-START | /infrastructure/test262/strict-error.test262.strict.html
16:55.22 INFO Closing window 8859b625-aa5f-4283-955f-463f85f94438
Closing window 8859b625-aa5f-4283-955f-463f85f94438
TEST-FAIL | /infrastructure/test262/strict-error.test262.strict.html | Test - ReferenceError: foo is not defined
window.test262HarnessDone<@http://web-platform.test:8000/infrastructure/test262/strict-error.test262.strict.html:12:15
Test.prototype.step@http://web-platform.test:8000/resources/testharness.js:2869:25
Test.prototype.step_func_done/<@http://web-platform.test:8000/resources/testharness.js:2944:32
done@http://web-platform.test:8000/resources/test262/test262-reporter.js:99:22
on_error@http://web-platform.test:8000/resources/test262/test262-reporter.js:156:9
16:55.63 TEST_END: Test OK. Subtests passed 0/1. Unexpected 0
TEST-OK | /infrastructure/test262/strict-error.test262.strict.html | took 420ms
16:55.63 TEST_START: /infrastructure/test262/strict-failure.test262.strict.html
TEST-START | /infrastructure/test262/strict-failure.test262.strict.html
16:55.64 INFO Closing window 4bcac05e-2ed8-4ad4-83ee-61efeb8e29be
Closing window 4bcac05e-2ed8-4ad4-83ee-61efeb8e29be
TEST-FAIL | /infrastructure/test262/strict-failure.test262.strict.html | Test - One should be two in strict mode Expected SameValue(«1», «2») to be true
window.test262HarnessDone<@http://web-platform.test:8000/infrastructure/test262/strict-failure.test262.strict.html:10:15
Test.prototype.step@http://web-platform.test:8000/resources/testharness.js:2869:25
Test.prototype.step_func_done/<@http://web-platform.test:8000/resources/testharness.js:2944:32
done@http://web-platform.test:8000/resources/test262/test262-reporter.js:99:22
on_error@http://web-platform.test:8000/resources/test262/test262-reporter.js:156:9
16:56.04 TEST_END: Test OK. Subtests passed 0/1. Unexpected 0
TEST-OK | /infrastructure/test262/strict-failure.test262.strict.html | took 412ms
16:56.04 TEST_START: /infrastructure/test262/strict.test262.html
TEST-START | /infrastructure/test262/strict.test262.html
16:56.05 INFO Closing window fadf3ca6-7d03-4b1a-a577-c00d86bb153e
Closing window fadf3ca6-7d03-4b1a-a577-c00d86bb153e
TEST-PASS | /infrastructure/test262/strict.test262.html | Test
16:56.49 TEST_END: Test OK. Subtests passed 1/1. Unexpected 0
TEST-OK | /infrastructure/test262/strict.test262.html | took 446ms
16:56.49 TEST_START: /infrastructure/test262/timeout.test262.html
TEST-START | /infrastructure/test262/timeout.test262.html
16:56.50 INFO Closing window f178fea0-2276-4468-9e6d-de73ed256cfe
Closing window f178fea0-2276-4468-9e6d-de73ed256cfe
TEST-NOTRUN | /infrastructure/test262/timeout.test262.html | Test
17:36.75 TEST_END: Test TIMEOUT. Subtests passed 0/1. Unexpected 0
TEST-TIMEOUT | /infrastructure/test262/timeout.test262.html | took 40259ms
17:36.75 TEST_START: /infrastructure/test262/unexpected-error.test262.html
TEST-START | /infrastructure/test262/unexpected-error.test262.html
17:36.76 INFO Closing window 99d53a13-dd0f-4720-866a-4aa090c08f33
Closing window 99d53a13-dd0f-4720-866a-4aa090c08f33
TEST-FAIL | /infrastructure/test262/unexpected-error.test262.html | Test - ReferenceError: foo is not defined
window.test262HarnessDone<@http://web-platform.test:8000/infrastructure/test262/unexpected-error.test262.html:12:15
Test.prototype.step@http://web-platform.test:8000/resources/testharness.js:2869:25
Test.prototype.step_func_done/<@http://web-platform.test:8000/resources/testharness.js:2944:32
done@http://web-platform.test:8000/resources/test262/test262-reporter.js:99:22
on_error@http://web-platform.test:8000/resources/test262/test262-reporter.js:156:9
17:37.18 TEST_END: Test OK. Subtests passed 0/1. Unexpected 0
TEST-OK | /infrastructure/test262/unexpected-error.test262.html | took 437ms
17:37.18 INFO No more tests
No more tests
17:37.23 INFO Closing logging queue
Closing logging queue
17:37.69 INFO STDOUT: cleanup aborted: args: /home/test/web-platform-tests/_venv3/android/android-sdk-linux/platform-tools/adb wait-for-device remount, exitcode: 1, stdout: Device must be bootloader unlocked
STDOUT: cleanup aborted: args: /home/test/web-platform-tests/_venv3/android/android-sdk-linux/platform-tools/adb wait-for-device remount, exitcode: 1, stdout: Device must be bootloader unlocked
17:37.23 INFO queue closed
queue closed
17:38.10 INFO STDOUT: cleanup aborted: args: /home/test/web-platform-tests/_venv3/android/android-sdk-linux/platform-tools/adb wait-for-device remount, exitcode: 1, stdout: Device must be bootloader unlocked
STDOUT: cleanup aborted: args: /home/test/web-platform-tests/_venv3/android/android-sdk-linux/platform-tools/adb wait-for-device remount, exitcode: 1, stdout: Device must be bootloader unlocked
17:38.10 SUITE_END

web-platform-test
~~~~~~~~~~~~~~~~~
Ran 444 checks (243 subtests, 201 tests)
Expected results: 436 (1 known intermittents)
Skipped: 4 tests
Unexpected results: 4
  subtest: 4 (4 fail)

Known Intermittent Results
--------------------------
/infrastructure/testdriver/actions/elementTiming.html
  KNOWN-INTERMITTENT-FAIL TestDriver actions: element timing - assert_array_equals: expected property 1 to be 150 but got 149 (expected array [50, 150] got [50, 149])
@http://web-platform.test:8000/infrastructure/testdriver/actions/elementTiming.html:67:22

Error Summary
-------------
/infrastructure/testdriver/virtual_authenticator.html
  FAIL Can add a credential - promise_test: Unhandled rejection with value: "error: Action add_credential failed: TypeError: credentials is undefined\nstacktrace:\n\twebAuthn_addCredential@chrome://remote/content/marionette/driver.sys.mjs:3782:9\n\tdispatch@chrome://remote/content/marionette/server.sys.mjs:317:49\n\texecute@chrome://remote/content/marionette/server.sys.mjs:289:16\n\tonPacket/<@chrome://remote/content/marionette/server.sys.mjs:262:20\n\tonPacket@chrome://remote/content/marionette/server.sys.mjs:263:9\n\t_onJSONObjectReady/<@chrome://remote/content/marionette/transport.sys.mjs:494:20\n"
  FAIL Can get the credentials - assert_equals: expected 1 but got 0
@http://web-platform.test:8000/infrastructure/testdriver/virtual_authenticator.html:49:16
  FAIL Can remove a credential - promise_test: Unhandled rejection with value: "error: Action remove_credential failed: [Exception... \"Component returned failure code: 0x80070057 (NS_ERROR_ILLEGAL_VALUE) [nsIWebAuthnService.removeCredential]\"  nsresult: \"0x80070057 (NS_ERROR_ILLEGAL_VALUE)\"  location: \"JS frame :: chrome://remote/content/marionette/webauthn.sys.mjs :: webauthn.removeCredential :: line 158\"  data: no]\nstacktrace:\n\twebauthn.removeCredential@chrome://remote/content/marionette/webauthn.sys.mjs:158:24\n\twebAuthn_removeCredential@chrome://remote/content/marionette/driver.sys.mjs:3960:19\n\tdispatch@chrome://remote/content/marionette/server.sys.mjs:317:49\n\texecute@chrome://remote/content/marionette/server.sys.mjs:289:16\n\tonPacket/<@chrome://remote/content/marionette/server.sys.mjs:262:20\n\tonPacket@chrome://remote/content/marionette/server.sys.mjs:263:9\n\t_onJSONObjectReady/<@chrome://remote/content/marionette/transport.sys.mjs:494:20\n"
  FAIL Can remove all credentials - promise_test: Unhandled rejection with value: "error: Action add_credential failed: TypeError: credentials is undefined\nstacktrace:\n\twebAuthn_addCredential@chrome://remote/content/marionette/driver.sys.mjs:3782:9\n\tdispatch@chrome://remote/content/marionette/server.sys.mjs:317:49\n\texecute@chrome://remote/content/marionette/server.sys.mjs:289:16\n\tonPacket/<@chrome://remote/content/marionette/server.sys.mjs:262:20\n\tonPacket@chrome://remote/content/marionette/server.sys.mjs:263:9\n\t_onJSONObjectReady/<@chrome://remote/content/marionette/transport.sys.mjs:494:20\n"
SUITE-END | took 783s
17:38.11 INFO Got 1 unexpected results, with 0 unexpected passes
Got 1 unexpected results, with 0 unexpected passes
17:38.11 wptserve INFO Stopped http server on 127.0.0.1:8000
wptserve Stopped http server on 127.0.0.1:8000
17:38.12 wptserve INFO Stopped http server on 127.0.0.1:8002
wptserve Stopped http server on 127.0.0.1:8002
17:38.12 wptserve INFO Stopped http server on 127.0.0.1:8003
wptserve Stopped http server on 127.0.0.1:8003
17:38.12 wptserve INFO Stopped http server on 127.0.0.1:8001
wptserve Stopped http server on 127.0.0.1:8001
17:38.12 wptserve INFO Stopped http server on 127.0.0.1:8443
wptserve Stopped http server on 127.0.0.1:8443
17:38.13 wptserve INFO Stopped http server on 127.0.0.1:8445
wptserve Stopped http server on 127.0.0.1:8445
17:38.14 wptserve INFO Stopped http server on 127.0.0.1:8446
wptserve Stopped http server on 127.0.0.1:8446
17:38.14 wptserve INFO Stopped http server on 127.0.0.1:9000
wptserve Stopped http server on 127.0.0.1:9000
17:38.14 wptserve INFO Stopped http server on 127.0.0.1:8444
wptserve Stopped http server on 127.0.0.1:8444
17:38.34 wptserve INFO Close on: (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('127.0.0.1', 8889))
wptserve Close on: (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('127.0.0.1', 8889))
17:38.59 wptserve INFO Close on: (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('127.0.0.1', 8888))
wptserve Close on: (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('127.0.0.1', 8888))
17:38.65 INFO Removed font: Ahem.ttf
Removed font: Ahem.ttf
17:38.67 INFO Closing logging queue
Closing logging queue
17:38.67 INFO queue closed
queue closed
[taskcluster 2026-04-28 20:42:15.066Z] === Task Finished ===
[taskcluster 2026-04-28 20:42:17.082Z] Unsuccessful task run with exit code: 1 completed in 1260.37 seconds
View more details on Community-TC Integration