Create new test type aamtest for accessibility API testing

[spectranaut]

This PR adds a new test type to test accessibility APIs exposed by browsers, as defined by the ARIA, Core-AAM and HTML-AAM specifications. The now merged RFC can be found here.

This is a replacement for: #53733

Instead of extending testharness, I added a new test type (aamtest) that is similar to wdspec tests and uses a lot of the same infrastructure. This idea came from @foolip in this comment on the RFC, and I think it looks good!

Here is a PR that adds documentation for this test type: #58632

To run tests on Linux:

On Debian distros:
apt install libatspi2.0-dev libcairo2-dev libgirepository1.0-dev

# Chrome
./wpt run chrome core-aam/aamtests/role/blockquote.py

# Chromium
./wpt run  --binary <chromiumbinary> chromium core-aam/aamtests/role/blockquote.py

# Firefox (needs --no-headless explicitly set)
./wpt run --no-headless firefox core-aam/aamtests/role/blockquote.py

On Mac:

Run chrome tests with --no-headless. Safari does not yet support this test type.

[community-tc-integration]

Uh oh! Looks like an error!

Client ID static/taskcluster/github does not have sufficient scopes and is missing the following scopes:

{
  "AnyOf": [
    "queue:rerun-task:taskcluster-github/RBJqfU0pQ82PBMqZMXxGYw/RqnO5x7vQu-LPaJ5z8Ue5w",
    "queue:rerun-task-in-project:none",
    {
      "AllOf": [
        "queue:rerun-task",
        "assume:scheduler-id:taskcluster-github/RBJqfU0pQ82PBMqZMXxGYw"
      ]
    }
  ]
}

This request requires the client to satisfy the following scope expression:

{
  "AnyOf": [
    "queue:rerun-task:taskcluster-github/RBJqfU0pQ82PBMqZMXxGYw/RqnO5x7vQu-LPaJ5z8Ue5w",
    "queue:rerun-task-in-project:none",
    {
      "AllOf": [
        "queue:rerun-task",
        "assume:scheduler-id:taskcluster-github/RBJqfU0pQ82PBMqZMXxGYw"
      ]
    }
  ]
}

---

• method: rerunTask
• errorCode: InsufficientScopes
• statusCode: 403
• time: 2026-02-11T20:24:29.184Z

[spectranaut]

@jcsteh -- I'd love your early feedback on this completely new direction to add AAM tests, the tests are like the wpt's webdriver spec tests, all in python!

Look at the blockquote test.

The APIs are passed to the test as arguments ("fixtures" in pytest speak -- defined in wai-aria-aam/support/fixtures_a11y_api.py). The atspi argument is a AtspiWrapper object, and the axapi argument is a AxapiWrapper object, and the ia2 argument is a Ia2Wrapper object.

You can see these tests already in the wpt.fyi for this PR: https://wpt.fyi/results/?label=pr_head&max-count=1&pr=57696

[jcsteh]

@spectranaut Thanks for the early ping and for your work on this. This looks really neat!

I haven't looked at this in-depth yet, but here are some early thoughts:

1. I notice that this moves away from the declarative approach and is more imperative. On one hand, that's what I was advocating for, so that's nice for me. :) On the other hand, I recall you feeling strongly about declarative tests, so I'd love to understand why you feel the imperative approach works with this Python framework, but didn't fit for the TestDriver framework. I totally understand if you just had a change of heart and incorporated that here, but if there's a different and/or another reason you think it makes more sense here, that understanding might guide other thinking and future possibilities.
2. This more or less flips the flow. Instead of writing web stuff and calling out to Python to test it, we now call out to the browser from Python to load web stuff and then test it in Python. At the risk of stating the obvious, while there were challenges with the former approach for complex cases (e.g. needing to potentially send Python code to be evaluated), there are also challenges with this latter approach for complex cases (e.g. testing mutations will require sending JS to be evaluated). It's probably fair to say that we need to run more Python than we do JS for these tests, so driving them with Python will reduce the amount of ugly cross-language shenanigans, but I just want to flag that we're not going to escape this altogether; we will absolutely need to test many kinds of mutations going forward. I reckon most of the obscurest browser engine bugs I have to fix end up being related to mutations in some way or another. :)
3. I'm not super familiar with this framework, so just to double check, is it definitely possible for us to evaluate whatever JS we need to run via the session object?
4. Can we await results from JS too; e.g. await some DOM event before executing something in Python? We'll mostly want to wait for accessibility events, not DOM events, but there are complex cases where being able to wait for some DOM event can be useful.
5. Speaking of accessibility events, I do think this will make supporting those simpler. We could have done that by sending Python from JS, having the Python block and then return the result to JS, which is what the Gecko IA2/ATK/UIA tests do. However, that might have been a bit tricky/ugly with TestDriver, whereas it's cleaner and more straightforward if we can keep it all in Python. FWIW, I wrote Python helpers to wait for specific IA2, ATK and UIA events for Gecko tests, so that should hopefully be helpful when we get to that point.
6. It'd be nice if we could avoid the if not atspi: return style boilerplate at the top of every test, but we probably can't. I thought about using a decorator that could handle this for us, but I suspect the "magic" used by fixtures wouldn't like that much and it only really reduces the boilerplate by 1 or 2 lines anyway (a decorator still means a line of code).

[spectranaut]

@jcsteh thanks as always for the thoughts! :)

On 1, imperative vs declarative -- tbh I never had a strong preference either way, maybe slight :) I think the declarative approach aligns the way the mapping of the Core-AAM are presented.. they are somewhat simplified and kind of have their own language for describing the APIs. Plus we can reuse all the manual tests Joanie maintained. But I think I've been convinced by you that closer to the API/imperative tests will get us better results -- and make a better and more flexible test suite in the long run.

On 2, on the python vs html+js flip -- yeah I see the tradeoffs! The tests in this PR all have inline html, but for more complicated tests, we can create an separate html file to open. I think if we are going to write imperative tests (which I've been convinced), I think we should write the tests in python, and choose those tradeoffs.

On 3, on session objects/executing javascript -- the session object is an implementation of webdriver maintained in wpt here: https://github.com/web-platform-tests/wpt/tree/master/tools/webdriver These tests have all of webdriver available to them, including the ability to send in javascript to execute, or sending clicks, keys, etc.

On 4, on DOM events -- in webdriver classic, you can't wait on DOM events, you can only poll for changes, which is probably good enough? There is a way to wait for things with webdriver bidi, but Safari doesn't have support for bidi yet.

On 5 accessibility events -- awesome, yes, that will be helpful, and I think accessibility event testing will be easier here too.

On 6, the if not atspi: return -- it's not great and I'll keep an eye out for options.. not sure that fixtures can help, but maybe some other pytest thing. I really want there to be a "not applicable" concept which can be applied to subtests, but I haven't dug into that.

[spectranaut]

Hi @jcsteh -- I'm noticing that these tests are flakey on Firefox on Linux.. and I wonder if you know why or can think of an easy fix. The flakes were caught by the Community-TC Integration / wpt-firefox-nightly-stability and are easy to reproduce locally.

Basically, the nodes all appear in the tree, but not all the correct attributes are set by the time we query for them.

In the code, before we run the test, we (1) load the webpage, then (2) find for the correct tab (role: document web), then (3) wait until "busy" is not set.

But when you run the test immediately after that, finding the node by DOM ID fails sometimes -- the blockquote node does not always have a DOM ID attribute. I added a poll to try to solve for this but it doesn't seem like a great solution, and then, I'm getting flakey failures while looking for another attribute in another test, as you can see in this CI report.

Am I waiting for busy on the wrong thing? Or is this bug in firefox?

[jcsteh]

Ah, this is due to caching granularity. By default, we only enable a small set of cached attributes to improve memory usage and performance, since a lot of clients don't need everything. When a client first requests something that isn't in the cache, we asynchronously enable it from that point forward. You can work around this by setting the pref accessibility.enable_all_cache_domains to true, the same way you set the accessibility.force_disabled pref.

[spectranaut]

Bad news, @jcsteh 😢
I turned on caching and I still see the flake. I confirmed the setting was on in about:config. See the flake report when I remove polling for the dom id and the flake report with polling enabled -- it's essentially the same as if this setting was not set.

[jcsteh]

Very odd. I'll need to get this running locally so I can shove some logging into Gecko and see what's going on. What's really strange is that we have a whole bunch of Gecko tests which cover exactly this behaviour.

[jcsteh]

@spectranaut, are you far enough along with Windows or Mac testing to know whether this flake shows up for Firefox on either of those platforms? That is, is this just a Linux flake at this stage or is that not conclusive yet?

[spectranaut]

@jcsteh Yes actually it is far enough along to test on mac -- I just tried: I ran the blockquote test on mac with the polling for dom id removed (the mac code also polls for dom id) and with the firefox setting accessibility.enable_all_cache_domains on and off. With the setting off, it failed to find the dom id on the node 6 out of 200 runs. With the setting on, it failed to find the dom id on the node 1 out of 200 runs. So less flakey but still flakey in both cases!

As for windows, I can also run the blockquote test there -- or thought I could. I can run it on windows against Chrome but it seems like wpt is failing (even other tests) when I tried to run against Firefox... It hangs when trying to start firefox using marionettedriver, I'm not sure why, didn't have that much time to figure it out.

[spectranaut]

hi @jonathan-j-lee ! Could you take a look at this alternative test format for the same AAM tests? Some of the code is the same and that code includes your review feedback on the other PR (#53733)

[spectranaut]

I made issues for the unrelated CI failures:

• test-jobs / infrastructure/ tests: macOS / 1 (of 1): macOS CI failure: /infrastructure/server/webtransport-h3 #59495
• infrastructure/ tests (firefox_android): Firefox CI failures: /infrastructure/testdriver/virtual_authenticator.html  #59493
• infrastructure/ tests (firefox): Firefox CI failures: /infrastructure/testdriver/virtual_authenticator.html  #59493

And after talking to @jcsteh I opened issues for Firefox related to the flakes:

• Flakey new WPT tests: properties not always available in accessibility API
• Flakey new WPT tests: Firefox not always available in Accessibility API

I sounds like it would be easier to debug these issues with Firefox after these tests land.

[jcsteh]

I sounds like it would be easier to debug these issues with Firefox after these tests land.

This seems reasonable to me. It does seem like these issues might be more in Gecko code than the tests or harness. Either way, we can follow up with fixes, working together if necessary.