diff --git a/docs/test-suite-design.md b/docs/test-suite-design.md index 6a104e2f1d42fd..c9d48c015f55ae 100644 --- a/docs/test-suite-design.md +++ b/docs/test-suite-design.md @@ -68,6 +68,10 @@ expectations: * [wdspec][] tests are written in Python and test [the WebDriver browser automation protocol](https://w3c.github.io/webdriver/) +* [aamtest][] tests are written in Python and test accessibility API mappings, + such as [Core Accessibility API Mappings](https://w3c.github.io/core-aam/) + or [HTML Accessibility API Mappings](https://w3c.github.io/html-aam/). + * [Manual tests][manual] rely on a human to run them and determine their result. diff --git a/docs/writing-tests/aamtest.md b/docs/writing-tests/aamtest.md new file mode 100644 index 00000000000000..58214e3e001aeb --- /dev/null +++ b/docs/writing-tests/aamtest.md @@ -0,0 +1,132 @@ +# aamtest tests + +The aamtest tests are used to verify the mapping of web content to +browser-exposed platform-specific accessibility APIs. These mappings are +specified by the ARIA working group of the W3C in the following specifications: + +* [Core-AAM](https://w3c.github.io/core-aam) +* [HTML-AAM](https://w3c.github.io/html-aam) +* [DPUB-AAM](https://w3c.github.io/dpub-aam) +* [MathML-AAM](https://w3c.github.io/mathml-aam) +* [Graphics-AAM](https://w3c.github.io/graphics-aam) +* [SVG-AAM](https://w3c.github.io/svg-aam/) + +These tests are written in [the Python programming +language](https://www.python.org/) and structured with [the pytest testing +framework](https://docs.pytest.org/en/latest/). + +The aamtest type is built on the [wdspec](wdspec) test type, and has access to +all the python fixtures defined for wdspec tests. It uses the web-platform-tests +maintained WebDriver client library to load HTML and to send other WebDriver +commands to the browser. + +The `wptrunner` will know a python file is an aamtest if it is contained within +an `aamtests` directory. + +## Platform-Specific Accessibility APIs + +Accessibility APIs are platform specific, each platform has their own API +(sometimes more than one). Assistive technologies, such as screen readers, +interact with the browser on behalf of a user via these APIs. The AAM +specifications explain how to expose web content through these APIs. You can +read more about [the APIs in +Core-AAM](https://w3c.github.io/core-aam/#intro_aapi). + +The table below lists: + +* APIs supported by the aamtest framework +* The name of the pytest fixture that returns access to the API (if you are + on the correct platform). +* The platform of that API. +* The python library that provides bindings to query the API. + +| API Name | Fixture Name | Platform | Python Bindings | +|---|---|---|---| +| Accessibility Toolkit ([ATK](https://developer.gnome.org/atk/stable/)) and Assistive Technology Service Provider Interface ([AT-SPI](https://gnome.pages.gitlab.gnome.org/at-spi2-core/libatspi/)) | `atspi` | Linux | [Provided through PyGObject](https://lazka.github.io/pgi-docs/#Atspi-2.0) | +| The NSAccessibility Protocol for macOS ([AX API](https://developer.apple.com/documentation/appkit/nsaccessibility)) | `axapi` | macOS | [pyobjc-framework-Accessibility](https://pypi.org/project/pyobjc-framework-Accessibility/) | +| MSAA with IAccessible2 1.3 ([IA2](https://wiki.linuxfoundation.org/accessibility/iaccessible2/start)) | `ia2` | Windows | Loading module [ia2_api_all.idl](https://github.com/LinuxA11y/IAccessible2) with [comtypes](https://pypi.org/project/comtypes/) | + +The APIs are exposed through a pytest fixture with the name in the table +above. The pytest fixture returns a wrapped version of the API. Requesting this +fixture on a platform where it is not supported will result in a `MISSING` +subtest. It is expected that each test file run on a given platform will have +one or more subtests that run to completion, as well as several subtests who's +results will not be recorded. This is because each test file should show how the +**same markup** is exposed in each supporting accessibility APIs/platforms. + +### Package dependencies for Linux API AT-SPI Python Bindings + +In order to test the Linux API AT-SPI, you need to have the following packages +installed: + +``` +sudo apt install libatspi2.0-dev libcairo2-dev libgirepository1.0-dev +``` + +## Adding new tests + +If you would like to add a new aamtest to a specification that does not yet have +coverage, add the tests to an `aamtests` subfolder. This subfolder indicates the +python files within it will be run as an aamtest. This includes restarting the +browser with accessibility enabled, if you are doing a full run of the test +suite. + +In the `aamtests` subfolder, the `conftest.py` will need to add the +`webdriver/test/support` path and `core-aam/aamtests/support` path to the +sys.path and add the paths as `pytest_plugins` in order to have access to the +appropriate fixtures. See `core-aam/aamtests/conftest.py`. + +### Test design + +Similar to [testharness.js](testharness) tests, each file is a test, and a +function that begins with the name "test_" is a subtest of that test. + +A typical test file contains some html markup and several subtests. Each subtest +will test how that markup is exposed in a single accessibility API. For example, +if you are testing how `
foobar widget
` is exposed in +accessibility APIs, and foobars are supported in the Linux API AT-SPI and macOS +API AX API, you should add a subtest called `test_atspi` and `test_axapi`, +respectively. Both subtests will load the same HTML, then query their respective +accessibility APIs. The subtest name should include, at least, the name of the +API being tested for ease of understanding the test results. + +For example, the file `foobar.py`: +```python +TEST_HTML = "
" + +# Test of the Linux accessibility API: AT-SPI +def test_atspi(atspi, session, inline): + # The `session` and `inline` fixtures are provided from the `wdspec` test infrastructure. + session.url = inline(TEST_HTML) + + # The `atspi` fixture wraps the AT-SPI python bindings and provides some helper functions, + # such as `find_node`, which finds a node by DOM ID. + node = atspi.find_node("test", session.url) + assert atspi.Accessible.get_role(node) == atspi.Role.FOOBAR + +# Test of the macOS accessibility API: AX API +def test_axapi(axapi, session, inline): + # The `session` and `inline` fixtures are provided from the `wdspec` test infrastructure. + session.url = inline(TEST_HTML) + + # The `axapi` fixture wraps the AX API python bindings and provides some helper functions, + # such as `find_node`, which finds a node by DOM ID. + node = axapi.find_node("test", session.url) + role = axapi.AXUIElementCopyAttributeValue(node, "AXRole", None)[1] + assert role == "AXFoobar" +``` + +## Adding support for an unsupported API + +To add an unsupported API: + +1. Add the python package that provides the python bindings for that API to + `tools/wptrunner/requirements_platform_accessibility.txt`. +2. Create a wrapper object for the new API in `newapi_wrapper.py` in + `core-aam/aamtests/support/`. It must inherit from `ApiWrapper` and follow + the same conventions as the other API wrappers, as appropriate. +3. Add the fixture for that API in + `core-aam/aamtests/support/fixtures_a11y_api.py`. +4. Update the table in the "Platform-Specific Accessibility APIs" section of this + document. +5. Add a new subtest to all the files that contain markup you would like to test.