Skip to content

Create new test type aamtest for accessibility API testing #57696

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
spectranaut wants to merge 38 commits into
base: master
Choose a base branch
from

Fix tables and links

d4fca40
Select commit
Loading
Failed to load commit list.
Open

Create new test type aamtest for accessibility API testing #57696

Fix tables and links
d4fca40
Select commit
Loading
Failed to load commit list.
Community-TC Integration / wpt-firefox-nightly-stability failed Apr 28, 2026 in 21m 8s

Community-TC (pull_request)

Verify that all tests affected by a pull request are stable when executed in firefox.

Details

View task in Taskcluster | View logs in Taskcluster | View task group in Taskcluster

Task Status

Started: 2026-04-28T20:23:11.842Z
Resolved: 2026-04-28T20:29:24.768Z
Task Execution Time: 6 minutes, 12 seconds, 926 milliseconds
Task Status: failed
Reason Resolved: failed
TaskId: SVHUiDJ4Q9O1waWc2Z45nQ
RunId: 0

Artifacts

- public/logs/live_backing.log
- public/logs/live.log
- public/results/checkrun.md

WPT Command: python3 ./wpt run --channel=nightly --verify --verify-no-chaos-mode --verify-repeat-loop=0 --verify-repeat-restart=10 --github-checks-text-file=/home/test/artifacts/checkrun.md --affected base_head --log-mach-level=info --log-mach=- -y --no-pause --no-restart-on-unexpected --install-fonts --verify-log-full --no-headless --binary=/home/test/build/firefox/firefox firefox

Some affected tests had inconsistent (flaky) results:

Unstable results

Test Subtest Results Messages
/core-aam/aamtests/attribute/aria_autocomplete_inline_list_both.py test_atspi[both] ERROR: 1/10, PASS: 9/10 setup error: Exception: Couldn't find browser firefox in accessibility API ATSPI.
/core-aam/aamtests/attribute/aria_autocomplete_inline_list_both.py test_atspi[inline] ERROR: 1/10, PASS: 9/10 setup error: Exception: Couldn't find browser firefox in accessibility API ATSPI.
/core-aam/aamtests/attribute/aria_autocomplete_inline_list_both.py test_atspi[list] ERROR: 1/10, PASS: 9/10 setup error: Exception: Couldn't find browser firefox in accessibility API ATSPI.

These may be pre-existing or new flakes. Please try to reproduce (see the above WPT command, though some flags may not be needed when running locally) and determine if your change introduced the flake. If you are unable to reproduce the problem, please tag @web-platform-tests/wpt-core-team in a comment for help.

[taskcluster 2026-04-28 20:23:11.906Z] Task ID: SVHUiDJ4Q9O1waWc2Z45nQ
[taskcluster 2026-04-28 20:23:11.906Z] Worker ID: 7902391123616509143
[taskcluster 2026-04-28 20:23:11.906Z] Worker Group: us-east4-a
[taskcluster 2026-04-28 20:23:11.906Z] Worker Node Type: projects/757942385826/machineTypes/n2-standard-4
[taskcluster 2026-04-28 20:23:11.906Z] Worker Pool: proj-wpt/ci
[taskcluster 2026-04-28 20:23:11.906Z] Worker Version: 46.1.0
[taskcluster 2026-04-28 20:23:11.906Z] Public IP: 34.85.161.22
[taskcluster 2026-04-28 20:23:11.906Z] Hostname: proj-wpt-ci-td28tudrttetmntf-kht6q

b1e71f63927a - Started downloading
b1e71f63927a - Downloaded in 0.005 seconds
66587c81b81a - Started downloading
08c7e224535e - Started downloading
08c7e224535e - Downloaded in 0.008 seconds
66587c81b81a - Downloaded in 0.174 seconds
32bbaaf98e4c - Started downloading
4f4fb700ef54 - Started downloading
4f4fb700ef54 - Downloaded in 0 seconds
5b82b858ea7e - Started downloading
dd5257950bd4 - Started downloading

...(5181 lines hidden)...

E           Exception: Couldn't find browser firefox in accessibility API ATSPI.

pid        = 4016
product_name = 'firefox'
self       = <aamtests.support.atspi_wrapper.AtspiWrapper object at 0x7fa0a0988670>
timeout    = 15.0

core-aam/aamtests/support/api_wrapper.py:26: Exception
  ERROR test_atspi[inline] - setup error: Exception: Couldn't find browser firefox in accessibility API ATSPI.
session = <Session 3b9e4720-20c9-48a6-8dbf-79f24bb85e5e>, default_timeout = 15.0

    @pytest.fixture
    def atspi(session, default_timeout):
        if platform != "linux":
            pytest.skip("NOT_APPLICABLE")
    
        from .atspi_wrapper import AtspiWrapper
    
        pid, product_name = pid_from(session.capabilities)
>       return AtspiWrapper(pid, product_name, default_timeout)

AtspiWrapper = <class 'aamtests.support.atspi_wrapper.AtspiWrapper'>
default_timeout = 15.0
pid        = 4016
product_name = 'firefox'
session    = <Session 3b9e4720-20c9-48a6-8dbf-79f24bb85e5e>

core-aam/aamtests/support/fixtures_a11y_api.py:30: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <aamtests.support.atspi_wrapper.AtspiWrapper object at 0x7fa093d98b50>
pid = 4016, product_name = 'firefox', timeout = 15.0

    def __init__(self, pid: int, product_name: str, timeout: float) -> None:
        """Setup for accessibility API testing.
    
        :pid: The PID of the process which exposes the accessibility API.
        :product_name: The name of the browser, used to find the browser in the accessibility API.
        :timeout: The timeout the test harness has set for this test, local timeouts can be set based on it.
        """
        self.product_name: str = product_name
        self.pid: int = pid
        self.root: Optional[Any] = None
        self.document: Optional[ApiNode] = None
        self.test_url: Optional[str] = None
        self.timeout: float = timeout
    
        self.root = self._find_browser()
    
        if not self.root:
>           raise Exception(
                f"Couldn't find browser {self.product_name} in accessibility API {self.api_name}."
            )
E           Exception: Couldn't find browser firefox in accessibility API ATSPI.

pid        = 4016
product_name = 'firefox'
self       = <aamtests.support.atspi_wrapper.AtspiWrapper object at 0x7fa093d98b50>
timeout    = 15.0

core-aam/aamtests/support/api_wrapper.py:26: Exception
  ERROR test_atspi[list] - setup error: Exception: Couldn't find browser firefox in accessibility API ATSPI.
session = <Session 3b9e4720-20c9-48a6-8dbf-79f24bb85e5e>, default_timeout = 15.0

    @pytest.fixture
    def atspi(session, default_timeout):
        if platform != "linux":
            pytest.skip("NOT_APPLICABLE")
    
        from .atspi_wrapper import AtspiWrapper
    
        pid, product_name = pid_from(session.capabilities)
>       return AtspiWrapper(pid, product_name, default_timeout)

AtspiWrapper = <class 'aamtests.support.atspi_wrapper.AtspiWrapper'>
default_timeout = 15.0
pid        = 4016
product_name = 'firefox'
session    = <Session 3b9e4720-20c9-48a6-8dbf-79f24bb85e5e>

core-aam/aamtests/support/fixtures_a11y_api.py:30: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <aamtests.support.atspi_wrapper.AtspiWrapper object at 0x7fa0a0943760>
pid = 4016, product_name = 'firefox', timeout = 15.0

    def __init__(self, pid: int, product_name: str, timeout: float) -> None:
        """Setup for accessibility API testing.
    
        :pid: The PID of the process which exposes the accessibility API.
        :product_name: The name of the browser, used to find the browser in the accessibility API.
        :timeout: The timeout the test harness has set for this test, local timeouts can be set based on it.
        """
        self.product_name: str = product_name
        self.pid: int = pid
        self.root: Optional[Any] = None
        self.document: Optional[ApiNode] = None
        self.test_url: Optional[str] = None
        self.timeout: float = timeout
    
        self.root = self._find_browser()
    
        if not self.root:
>           raise Exception(
                f"Couldn't find browser {self.product_name} in accessibility API {self.api_name}."
            )
E           Exception: Couldn't find browser firefox in accessibility API ATSPI.

pid        = 4016
product_name = 'firefox'
self       = <aamtests.support.atspi_wrapper.AtspiWrapper object at 0x7fa0a0943760>
timeout    = 15.0

core-aam/aamtests/support/api_wrapper.py:26: Exception
 4:20.40 INFO Got 1 unexpected results, with 0 unexpected passes
 4:20.40 wptserve INFO Stopped http server on 127.0.0.1:8000
 4:20.40 wptserve INFO Stopped http server on 127.0.0.1:8002
 4:20.40 wptserve INFO Stopped http server on 127.0.0.1:8001
 4:20.40 wptserve INFO Stopped http server on 127.0.0.1:8003
 4:20.41 wptserve INFO Stopped http server on 127.0.0.1:8444
 4:20.41 wptserve INFO Stopped http server on 127.0.0.1:8446
 4:20.41 wptserve INFO Stopped http server on 127.0.0.1:8443
 4:20.41 wptserve INFO Stopped http server on 127.0.0.1:8445
 4:20.42 wptserve INFO Stopped http server on 127.0.0.1:9000
 4:20.42 wptserve INFO Stopped WebTransport over HTTP/3 server on 127.0.0.1:11000
 4:20.70 wptserve INFO Close on: (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('127.0.0.1', 8889))
 4:20.70 wptserve INFO Close on: (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('127.0.0.1', 8888))
 4:20.75 INFO Removed font: Ahem.ttf
 4:20.77 INFO Closing logging queue
 4:20.77 INFO queue closed
 4:20.80 INFO ::: Ran 10 of expected 10 iterations.
 4:20.80 INFO ## All results ##

 4:20.80 INFO ### /core-aam/aamtests/attribute/aria_autocomplete_inline_list_both.py ###
 4:20.80 INFO |       Subtest        |           Results           |                                       Messages                                      |
 4:20.80 INFO |----------------------|-----------------------------|-------------------------------------------------------------------------------------|
 4:20.80 INFO |                      | OK                          |                                                                                     |
 4:20.80 INFO | `test_atspi[both]`   | **ERROR: 1/10, PASS: 9/10** | `setup error: Exception: Couldn't find browser firefox in accessibility API ATSPI.` |
 4:20.80 INFO | `test_atspi[inline]` | **ERROR: 1/10, PASS: 9/10** | `setup error: Exception: Couldn't find browser firefox in accessibility API ATSPI.` |
 4:20.80 INFO | `test_atspi[list]`   | **ERROR: 1/10, PASS: 9/10** | `setup error: Exception: Couldn't find browser firefox in accessibility API ATSPI.` |
 4:20.80 INFO 
 4:20.80 INFO ### /core-aam/aamtests/attribute/aria_braillelabel.py ###
 4:20.80 INFO |   Subtest    | Results | Messages |
 4:20.80 INFO |--------------|---------|----------|
 4:20.80 INFO |              | OK      |          |
 4:20.80 INFO | `test_atspi` | PASS    |          |
 4:20.80 INFO 
 4:20.80 INFO ### /core-aam/aamtests/attribute/aria_error_message.py ###
 4:20.80 INFO |   Subtest    | Results | Messages |
 4:20.80 INFO |--------------|---------|----------|
 4:20.80 INFO |              | OK      |          |
 4:20.80 INFO | `test_atspi` | PASS    |          |
 4:20.80 INFO 
 4:20.80 INFO ### /core-aam/aamtests/role/blockquote.py ###
 4:20.80 INFO |   Subtest    | Results | Messages |
 4:20.80 INFO |--------------|---------|----------|
 4:20.80 INFO |              | OK      |          |
 4:20.80 INFO | `test_atspi` | PASS    |          |
 4:20.80 INFO 
 4:20.80 INFO ### /core-aam/aamtests/role/button.py ###
 4:20.80 INFO |                Subtest                | Results | Messages |
 4:20.80 INFO |---------------------------------------|---------|----------|
 4:20.80 INFO |                                       | OK      |          |
 4:20.80 INFO | `test_atspi[no-attributes]`           | PASS    |          |
 4:20.80 INFO | `test_atspi[aria-pressed-undefined]`  | PASS    |          |
 4:20.80 INFO | `test_atspi[aria-haspopup-undefined]` | PASS    |          |
 4:20.80 INFO | `test_atspi[aria-haspopup-false]`     | PASS    |          |
 4:20.80 INFO 
 4:20.80 INFO ### /core-aam/aamtests/role/button_haspopup.py ###
 4:20.80 INFO |        Subtest        | Results | Messages |
 4:20.80 INFO |-----------------------|---------|----------|
 4:20.80 INFO |                       | OK      |          |
 4:20.80 INFO | `test_atspi[true]`    | PASS    |          |
 4:20.80 INFO | `test_atspi[menu]`    | PASS    |          |
 4:20.80 INFO | `test_atspi[listbox]` | PASS    |          |
 4:20.80 INFO | `test_atspi[tree]`    | PASS    |          |
 4:20.80 INFO | `test_atspi[grid]`    | PASS    |          |
 4:20.80 INFO | `test_atspi[dialog]`  | PASS    |          |
 4:20.80 INFO 
 4:20.80 INFO ### /core-aam/aamtests/role/button_pressed.py ###
 4:20.80 INFO |       Subtest       | Results | Messages |
 4:20.80 INFO |---------------------|---------|----------|
 4:20.80 INFO |                     | OK      |          |
 4:20.80 INFO | `test_atspi[true]`  | PASS    |          |
 4:20.80 INFO | `test_atspi[false]` | PASS    |          |
 4:20.80 INFO 
 4:20.80 INFO ## Unstable results ##

 4:20.80 INFO |                                 Test                                 |       Subtest        |           Results           |                                       Messages                                      |
 4:20.80 INFO |----------------------------------------------------------------------|----------------------|-----------------------------|-------------------------------------------------------------------------------------|
 4:20.80 INFO | `/core-aam/aamtests/attribute/aria_autocomplete_inline_list_both.py` | `test_atspi[both]`   | **ERROR: 1/10, PASS: 9/10** | `setup error: Exception: Couldn't find browser firefox in accessibility API ATSPI.` |
 4:20.80 INFO | `/core-aam/aamtests/attribute/aria_autocomplete_inline_list_both.py` | `test_atspi[inline]` | **ERROR: 1/10, PASS: 9/10** | `setup error: Exception: Couldn't find browser firefox in accessibility API ATSPI.` |
 4:20.80 INFO | `/core-aam/aamtests/attribute/aria_autocomplete_inline_list_both.py` | `test_atspi[list]`   | **ERROR: 1/10, PASS: 9/10** | `setup error: Exception: Couldn't find browser firefox in accessibility API ATSPI.` |
 4:20.80 INFO 
 4:20.80 INFO ::: Running tests in a loop with restarts 10 times : FAIL
 4:20.80 INFO :::
 4:20.80 ERROR ::: Test verification FAIL
 4:20.80 INFO :::
[taskcluster 2026-04-28 20:29:23.509Z] === Task Finished ===
[taskcluster 2026-04-28 20:29:23.932Z] Unsuccessful task run with exit code: 1 completed in 372.027 seconds
View more details on Community-TC Integration