From 1ba37bad94a8ca995e993b3424a4813debef64c6 Mon Sep 17 00:00:00 2001 From: Valerie Young Date: Thu, 19 Mar 2026 17:20:49 -0700 Subject: [PATCH 1/4] Add documentation for aamtest test type --- docs/test-suite-design.md | 4 ++ docs/writing-tests/aamtest.md | 87 +++++++++++++++++++++++++++++++++++ 2 files changed, 91 insertions(+) create mode 100644 docs/writing-tests/aamtest.md diff --git a/docs/test-suite-design.md b/docs/test-suite-design.md index 6a104e2f1d42fd..057f352ad86945 100644 --- a/docs/test-suite-design.md +++ b/docs/test-suite-design.md @@ -68,6 +68,10 @@ expectations: * [wdspec][] tests are written in Python and test [the WebDriver browser automation protocol](https://w3c.github.io/webdriver/) +* [aamtest][] tests are written in Python and test accessibility API mappings, + such as [Core Accessibility API Mappings](https://w3c.github.io/core-aam/) + or [HTML Accessibility API Mappings](https://w3c.github.io/core-aam/). + * [Manual tests][manual] rely on a human to run them and determine their result. diff --git a/docs/writing-tests/aamtest.md b/docs/writing-tests/aamtest.md new file mode 100644 index 00000000000000..03e6270af04c0e --- /dev/null +++ b/docs/writing-tests/aamtest.md @@ -0,0 +1,87 @@ +# aamtest tests + +The aamtest tests are used to verify the mapping of web content to browser-exposed platform-specific accessibility APIs. These mappings are specified by the ARIA working group of the W3C in the following specifications: +* [Core-AAM](https://w3c.github.io/core-aam) +* [HTML-AAM](https://w3c.github.io/html-aam) +* [DPUB-AAM](https://w3c.github.io/dpub-aam) +* [MathML-AAM](https://w3c.github.io/mathml-aam) +* [Graphics-AAM](https://w3c.github.io/graphics-aam) +* [SVG-AAM](https://w3c.github.io/svg-aam/) + +These tests are written in [the Python programming language](https://www.python.org/) and structured with [the pytest testing framework](https://docs.pytest.org/en/latest/). + +The aamtest type is built on the [wdspec](wdspec) test type, and has access to all the python fixtures defined for wdspec tests. It uses the web-platform-tests maintained WebDriver client library to load HTML and to send other WebDriver commands to the browser. + +The `wptrunner` will know a python file is an aamtest if it is contained within an `aamtests` directory. + +## Platform-Specific Accessibility APIs + +Accessibility APIs are platform specific, each platform has their own API (sometimes more than one). Assistive technologies, such as screen readers, interact with the browser on behalf of a user via these APIs. The AAM specifications explain how to expose web content through these APIs. You can read more about [the APIs in Core-AAM](https://w3c.github.io/core-aam/#intro_aapi). + +The table below lists: +* APIs supported by the aamtest framework +* The name of the pytest fixture that returns access to the API (if you are on the correct platform). +* The platform of that API. +* The python library that provides bindings to query the API. + +| API Name | Fixture Name | Platform | Python Bindings | +|---|---|---|---| +| Accessibility Toolkit ([ATK](https://developer.gnome.org/atk/stable/)) and Assistive Technology Service Provider Interface ([AT-SPI](https://gnome.pages.gitlab.gnome.org/at-spi2-core/libatspi/)) | `atspi` | Linux | [Provided through PyGObject](https://lazka.github.io/pgi-docs/#Atspi-2.0) | +| The NSAccessibility Protocol for macOS ([AX API](https://developer.apple.com/documentation/appkit/nsaccessibility)) | `axapi` | macOS | [pyobjc-framework-Accessibility](https://pypi.org/project/pyobjc-framework-Accessibility/) | +| MSAA with IAccessible2 1.3 ([IA2](https://wiki.linuxfoundation.org/accessibility/iaccessible2/start)) | `ia2` | Windows | Loading module [ia2_api_all.idl](https://github.com/LinuxA11y/IAccessible2) with [comtypes](https://pypi.org/project/comtypes/) | + +The APIs are exposed through a pytest fixture with the name in the table above. The pytest fixture returns a wrapped version of the API. Requesting this fixture on a platform where it is not supported will result in a `PRECONDITION_FAILED` subtest result. It is expected that each test file run on a given platform will have one or more subtests that run to completion, as well as several subtests that result in `PRECONDITION_FAILED`. This is because each test file should show how the **same markup** is exposed in each supporting accessibility APIs/platforms. + +### Package dependencies for Linux API AT-SPI Python Bindings + +In order to test the Linux API AT-SPI, you need to have the following packages installed: +``` +sudo apt install libatspi2.0-dev libcairo2-dev libgirepository1.0-dev +``` + +## Adding new tests + +If you would like to add a new aamtest to a specification that does not yet have coverage, add the tests to an `aamtests` subfolder. This subfolder indicates the python files within it will be run as an aamtest. This includes restarting the browser with accessibility enabled, if you are doing a full run of the test suite. + +In the `aamtests` subfolder, the `conftest.py` will need to add the `webdriver/test/support` path and `core-aam/aamtests/support` path to the sys.path and add the paths as `pytest_plugins` in order to have access to the appropriate fixtures. See `core-aam/aamtests/conftest.py`. + +### Test design + +Similar to [testharness.js](testharness) tests, each file is a test, and a function that begins with the name "test_" is a subtest of that test. + +A typical test file contains some html markup and several subtests. Each subtest will test how that markup is exposed in a single accessibility API. For example, if you are testing how `
foobar widget
` is exposed in accessibility APIs, and foobars are supported in the Linux API AT-SPI and macOS API AX API, you should add a subtest called `test_atspi` and `test_axapi`, respectively. Both subtests will load the same HTML, then query their respective accessibility APIs. The subtest name should include, at least, the name of the API being tested for ease of understanding the test results. + +For example, the file `foobar.py`: +```python +TEST_HTML = "
" + +# Test of the Linux accessibility API: AT-SPI +def test_atspi(atspi, session, inline): + # The `session` and `inline` fixtures are provided from the `wdspec` test infrastructure. + session.url = inline(TEST_HTML) + + # The `atspi` fixture wraps the AT-SPI python bindings and provides some helper functions, + # such as `find_node`, which finds a node by DOM ID. + node = atspi.find_node("test", session.url) + assert atspi.Accessible.get_role(node) == atspi.Role.FOOBAR + +# Test of the macOS accessibility API: AX API +def test_axapi(axapi, session, inline): + # The `session` and `inline` fixtures are provided from the `wdspec` test infrastructure. + session.url = inline(TEST_HTML) + + # The `axapi` fixture wraps the AX API python bindings and provides some helper functions, + # such as `find_node`, which finds a node by DOM ID. + node = axapi.find_node("test", session.url) + role = axapi.AXUIElementCopyAttributeValue(node, "AXRole", None)[1] + assert role == "AXFoobar" +``` + +## Adding support for an unsupported API + +To add an unsupported API: +1. Add the python package that provides the python bindings for that API to `tools/wptrunner/requirements_platform_accessibility.txt`. +2. Create a wrapper object for the new API in `newapi_wrapper.py` in `core-aam/aamtests/support/`. It must inherit from `ApiWrapper` and follow the same conventions as the other API wrappers, as appropriate. +3. Add the fixture for that API in `core-aam/aamtests/support/fixtures_a11y_api.py`. +4. Update the table in the "Platform-Specific Accessibility APIs" section of this document. +5. Add a new subtest to all the files that contain markup you would like to test. From 96c6f97a32bc6570afc0070bcaa584a10bbbfce3 Mon Sep 17 00:00:00 2001 From: Valerie Young Date: Thu, 19 Mar 2026 17:46:12 -0700 Subject: [PATCH 2/4] Wrap at 80 --- docs/writing-tests/aamtest.md | 78 ++++++++++++++++++++++++++++------- 1 file changed, 62 insertions(+), 16 deletions(-) diff --git a/docs/writing-tests/aamtest.md b/docs/writing-tests/aamtest.md index 03e6270af04c0e..1251d019e2d883 100644 --- a/docs/writing-tests/aamtest.md +++ b/docs/writing-tests/aamtest.md @@ -1,6 +1,9 @@ # aamtest tests -The aamtest tests are used to verify the mapping of web content to browser-exposed platform-specific accessibility APIs. These mappings are specified by the ARIA working group of the W3C in the following specifications: +The aamtest tests are used to verify the mapping of web content to +browser-exposed platform-specific accessibility APIs. These mappings are +specified by the ARIA working group of the W3C in the following specifications: + * [Core-AAM](https://w3c.github.io/core-aam) * [HTML-AAM](https://w3c.github.io/html-aam) * [DPUB-AAM](https://w3c.github.io/dpub-aam) @@ -8,19 +11,32 @@ The aamtest tests are used to verify the mapping of web content to browser-expos * [Graphics-AAM](https://w3c.github.io/graphics-aam) * [SVG-AAM](https://w3c.github.io/svg-aam/) -These tests are written in [the Python programming language](https://www.python.org/) and structured with [the pytest testing framework](https://docs.pytest.org/en/latest/). +These tests are written in [the Python programming +language](https://www.python.org/) and structured with [the pytest testing +framework](https://docs.pytest.org/en/latest/). -The aamtest type is built on the [wdspec](wdspec) test type, and has access to all the python fixtures defined for wdspec tests. It uses the web-platform-tests maintained WebDriver client library to load HTML and to send other WebDriver commands to the browser. +The aamtest type is built on the [wdspec](wdspec) test type, and has access to +all the python fixtures defined for wdspec tests. It uses the web-platform-tests +maintained WebDriver client library to load HTML and to send other WebDriver +commands to the browser. -The `wptrunner` will know a python file is an aamtest if it is contained within an `aamtests` directory. +The `wptrunner` will know a python file is an aamtest if it is contained within +an `aamtests` directory. ## Platform-Specific Accessibility APIs -Accessibility APIs are platform specific, each platform has their own API (sometimes more than one). Assistive technologies, such as screen readers, interact with the browser on behalf of a user via these APIs. The AAM specifications explain how to expose web content through these APIs. You can read more about [the APIs in Core-AAM](https://w3c.github.io/core-aam/#intro_aapi). +Accessibility APIs are platform specific, each platform has their own API +(sometimes more than one). Assistive technologies, such as screen readers, +interact with the browser on behalf of a user via these APIs. The AAM +specifications explain how to expose web content through these APIs. You can +read more about [the APIs in +Core-AAM](https://w3c.github.io/core-aam/#intro_aapi). The table below lists: + * APIs supported by the aamtest framework -* The name of the pytest fixture that returns access to the API (if you are on the correct platform). +* The name of the pytest fixture that returns access to the API (if you are + on the correct platform). * The platform of that API. * The python library that provides bindings to query the API. @@ -30,26 +46,50 @@ The table below lists: | The NSAccessibility Protocol for macOS ([AX API](https://developer.apple.com/documentation/appkit/nsaccessibility)) | `axapi` | macOS | [pyobjc-framework-Accessibility](https://pypi.org/project/pyobjc-framework-Accessibility/) | | MSAA with IAccessible2 1.3 ([IA2](https://wiki.linuxfoundation.org/accessibility/iaccessible2/start)) | `ia2` | Windows | Loading module [ia2_api_all.idl](https://github.com/LinuxA11y/IAccessible2) with [comtypes](https://pypi.org/project/comtypes/) | -The APIs are exposed through a pytest fixture with the name in the table above. The pytest fixture returns a wrapped version of the API. Requesting this fixture on a platform where it is not supported will result in a `PRECONDITION_FAILED` subtest result. It is expected that each test file run on a given platform will have one or more subtests that run to completion, as well as several subtests that result in `PRECONDITION_FAILED`. This is because each test file should show how the **same markup** is exposed in each supporting accessibility APIs/platforms. +The APIs are exposed through a pytest fixture with the name in the table +above. The pytest fixture returns a wrapped version of the API. Requesting this +fixture on a platform where it is not supported will result in a +`PRECONDITION_FAILED` subtest result. It is expected that each test file run on +a given platform will have one or more subtests that run to completion, as well +as several subtests that result in `PRECONDITION_FAILED`. This is because each +test file should show how the **same markup** is exposed in each supporting +accessibility APIs/platforms. ### Package dependencies for Linux API AT-SPI Python Bindings -In order to test the Linux API AT-SPI, you need to have the following packages installed: +In order to test the Linux API AT-SPI, you need to have the following packages +installed: + ``` sudo apt install libatspi2.0-dev libcairo2-dev libgirepository1.0-dev ``` ## Adding new tests -If you would like to add a new aamtest to a specification that does not yet have coverage, add the tests to an `aamtests` subfolder. This subfolder indicates the python files within it will be run as an aamtest. This includes restarting the browser with accessibility enabled, if you are doing a full run of the test suite. +If you would like to add a new aamtest to a specification that does not yet have +coverage, add the tests to an `aamtests` subfolder. This subfolder indicates the +python files within it will be run as an aamtest. This includes restarting the +browser with accessibility enabled, if you are doing a full run of the test +suite. -In the `aamtests` subfolder, the `conftest.py` will need to add the `webdriver/test/support` path and `core-aam/aamtests/support` path to the sys.path and add the paths as `pytest_plugins` in order to have access to the appropriate fixtures. See `core-aam/aamtests/conftest.py`. +In the `aamtests` subfolder, the `conftest.py` will need to add the +`webdriver/test/support` path and `core-aam/aamtests/support` path to the +sys.path and add the paths as `pytest_plugins` in order to have access to the +appropriate fixtures. See `core-aam/aamtests/conftest.py`. ### Test design -Similar to [testharness.js](testharness) tests, each file is a test, and a function that begins with the name "test_" is a subtest of that test. +Similar to [testharness.js](testharness) tests, each file is a test, and a +function that begins with the name "test_" is a subtest of that test. -A typical test file contains some html markup and several subtests. Each subtest will test how that markup is exposed in a single accessibility API. For example, if you are testing how `
foobar widget
` is exposed in accessibility APIs, and foobars are supported in the Linux API AT-SPI and macOS API AX API, you should add a subtest called `test_atspi` and `test_axapi`, respectively. Both subtests will load the same HTML, then query their respective accessibility APIs. The subtest name should include, at least, the name of the API being tested for ease of understanding the test results. +A typical test file contains some html markup and several subtests. Each subtest +will test how that markup is exposed in a single accessibility API. For example, +if you are testing how `
foobar widget
` is exposed in +accessibility APIs, and foobars are supported in the Linux API AT-SPI and macOS +API AX API, you should add a subtest called `test_atspi` and `test_axapi`, +respectively. Both subtests will load the same HTML, then query their respective +accessibility APIs. The subtest name should include, at least, the name of the +API being tested for ease of understanding the test results. For example, the file `foobar.py`: ```python @@ -80,8 +120,14 @@ def test_axapi(axapi, session, inline): ## Adding support for an unsupported API To add an unsupported API: -1. Add the python package that provides the python bindings for that API to `tools/wptrunner/requirements_platform_accessibility.txt`. -2. Create a wrapper object for the new API in `newapi_wrapper.py` in `core-aam/aamtests/support/`. It must inherit from `ApiWrapper` and follow the same conventions as the other API wrappers, as appropriate. -3. Add the fixture for that API in `core-aam/aamtests/support/fixtures_a11y_api.py`. -4. Update the table in the "Platform-Specific Accessibility APIs" section of this document. + +1. Add the python package that provides the python bindings for that API to + `tools/wptrunner/requirements_platform_accessibility.txt`. +2. Create a wrapper object for the new API in `newapi_wrapper.py` in + `core-aam/aamtests/support/`. It must inherit from `ApiWrapper` and follow + the same conventions as the other API wrappers, as appropriate. +3. Add the fixture for that API in + `core-aam/aamtests/support/fixtures_a11y_api.py`. +4. Update the table in the "Platform-Specific Accessibility APIs" section of this + document. 5. Add a new subtest to all the files that contain markup you would like to test. From 56d7b55d21cd139c80eeea6c0e8550fa30ab05f3 Mon Sep 17 00:00:00 2001 From: Valerie Young Date: Thu, 16 Apr 2026 13:33:17 -0700 Subject: [PATCH 3/4] remove precondition failed reference --- docs/writing-tests/aamtest.md | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/docs/writing-tests/aamtest.md b/docs/writing-tests/aamtest.md index 1251d019e2d883..58214e3e001aeb 100644 --- a/docs/writing-tests/aamtest.md +++ b/docs/writing-tests/aamtest.md @@ -48,12 +48,11 @@ The table below lists: The APIs are exposed through a pytest fixture with the name in the table above. The pytest fixture returns a wrapped version of the API. Requesting this -fixture on a platform where it is not supported will result in a -`PRECONDITION_FAILED` subtest result. It is expected that each test file run on -a given platform will have one or more subtests that run to completion, as well -as several subtests that result in `PRECONDITION_FAILED`. This is because each -test file should show how the **same markup** is exposed in each supporting -accessibility APIs/platforms. +fixture on a platform where it is not supported will result in a `MISSING` +subtest. It is expected that each test file run on a given platform will have +one or more subtests that run to completion, as well as several subtests who's +results will not be recorded. This is because each test file should show how the +**same markup** is exposed in each supporting accessibility APIs/platforms. ### Package dependencies for Linux API AT-SPI Python Bindings From 6ec5a79a3b5b7c83312a3b53f78a6a1e0a177b75 Mon Sep 17 00:00:00 2001 From: Valerie Young Date: Mon, 27 Apr 2026 14:11:01 -0700 Subject: [PATCH 4/4] Correct link --- docs/test-suite-design.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/test-suite-design.md b/docs/test-suite-design.md index 057f352ad86945..c9d48c015f55ae 100644 --- a/docs/test-suite-design.md +++ b/docs/test-suite-design.md @@ -70,7 +70,7 @@ expectations: * [aamtest][] tests are written in Python and test accessibility API mappings, such as [Core Accessibility API Mappings](https://w3c.github.io/core-aam/) - or [HTML Accessibility API Mappings](https://w3c.github.io/core-aam/). + or [HTML Accessibility API Mappings](https://w3c.github.io/html-aam/). * [Manual tests][manual] rely on a human to run them and determine their result.