Extend testdriver.js and wptrunner to test AAMs (platform accessibility APIs) #53733
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,133 @@ | ||
| <!doctype html> | ||
| <meta charset=utf-8> | ||
| <html> | ||
| <head> | ||
| <title>aria-braillelabel</title> | ||
| <script src="/resources/testharness.js"></script> | ||
| <script src="/resources/testharnessreport.js"></script> | ||
| <script src="/resources/testdriver.js"></script> | ||
| <script src="/resources/testdriver-vendor.js"></script> | ||
| <script src="/resources/testdriver-actions.js"></script> | ||
| <script src="/core-aam/scripts/aam-utils.js"></script> | ||
| </head> | ||
| <body> | ||
|
|
||
| <p> | ||
| This test is marked tentative because it uses an experimental testing API | ||
| that is presently under development. | ||
| </p> | ||
|
|
||
| <input role='combobox' id='test1' aria-autocomplete='both'> | ||
| <input role='combobox' id='test2' aria-autocomplete='inline'> | ||
| <input role='combobox' id='test3' aria-autocomplete='list'> | ||
|
|
||
| <script> | ||
| AAMUtils.verifyAPI( | ||
| 'input[role=combobox][aria-autocomplete=both]', | ||
| 'test1', | ||
| { | ||
|
Contributor
There was a problem hiding this comment. For my knowledge, is this the format we expect all new tests to be written in, or is this an intermediate format to simply test the extension? Or something else entirely?
Contributor
Author
There was a problem hiding this comment. Good question -- this is the current proposed test format, but the format of the tests is still up for debate -- I'd love any feedback you have! However, feedback on the test format shouldn't block this PR, it's something we can change later and I expect to iterate on. From previous conversations (at last years TPAC, and informal), this is the current test design we've settled on. For some background: we also considered "bag of properties", where the backend passes back all the accessibility properties it can find for a node, and we considered writing tests in python (which the APIs are ultimately queried in) and sending the python back as a string, and executing it.
Contributor
There was a problem hiding this comment. You might also consider including the test parameters for a help function as data-expando attributes in the test markup... ARIAUtils does something like this. Currently you have a test case and a test call that have to align, but are separated sometimes by dozens of lines... in my experience, this is error prone. For example, someone could add the test case, but include an ID typo in the call, and the test looks like it passes but was a duplicate instead. Note that this is not a contrived example. It happened in some of the WPT tests, so we've added authoring changes and additional validation steps to help reduce or avoid this type of occurrence.
Contributor
Author
There was a problem hiding this comment. Thanks that is a good point, I'd like to revisit these tests with that in mind, but hopefully after the rest of this test infrastructure lands.
Contributor
There was a problem hiding this comment. I'm concerned that there are things we just can't test with declarative assertion statements like this. I've raised this before and advocated for the imperative approach of evaluating Python code. While I don't fully understand why, I trust that you and others have good reasons for choosing the approach in this PR over the Python approach, so I won't rehash that here. However, it would ease my concerns significantly if you could give me even a rough, high level outline of your thoughts on how we might expand this framework to test things like:
I can provide examples from Firefox tests if this would help understanding. None of this is well defined (if it's defined at all) in the AAMs right now, so we're quite some way off being able to test this anyway and we obviously don't need to work out the fine details right now. However, I just want to make sure that we aren't "backing ourselves into a corner" and landing on a solution which can't be extended to solve these problems without significant re-architecture and painful conversion of existing tests. I guess it wouldn't be difficult to add another method which evaluated Python code in the context of something provided by the various Impl classes and returned a serialised result. The question is whether you think that is an acceptable path for cases like this which don't fit into the assertion statement pattern. If not- i.e. the reasons for the choice of the assertion statement pattern preclude this option - then we're going to need another approach for those cases.
Contributor
Author
There was a problem hiding this comment. I appreciate you re-raising this point, @jcsteh. It was a long time after those discussions at TPAC that I had a chance to work on this project again, like a year, and I think I didn't look further into the python imperative approach because these declarative assertions are enough to test the AAMs as they are now -- and basically writing python as a string in an HTML file felt a bit awkward. Not to say this solution isn't awkward, in different ways. I'd really like to stay focused on finding a solution for the AAMs as they are written now, and find the shortest path to that (since even the shortest path is quite long) -- but I think it might be worth spending some time to actually prototype out what the "send python to the backend" tests and infrastructure would look like. The ability to test literally anything in the API exactly is a very motivating. You have sent me links to how mozilla does it and I can look at those again. Another stray thought about the difference between executing python directly or having some kind of declarative test language, like used in this PR -- at some point, rather than every test writing the same code to get all the relations of a node -- we are going to want to write test helpers/abstractions. So at what point do we want to do that? How easy do we want to make writing the same test "check that relation X of node with DOM ID Y refers to the node with DOM ID Z"? Which is to say, the abstractions in these tests might show up again in python helper functions. For completeness, here are some thoughts on your example future test cases:
In this test suite, when you get an accessibility object in the accessibility tree, it returns a DOM ID if it exists. For example, the test for aria-errormessage:
I could image something like
Ah unfortunately I'm too unfamiliar with this for an off the cuff suggestion. We can talk about it at TPAC?
Events are something we definitely want and would need their own API. I imaged some kind of "Record events on this node" sent to the backend, then you do whatever you want to trigger the event using WebDriver, then a "Check that event occurred" that will wait if it hasn't seen the event and at some point times out.
Contributor
There was a problem hiding this comment.
I am happy to provide specific examples as this conversation evolves so you don't have to dig too much.
That's a great point. They could either be Python helper functions or JS functions which construct and evaluate Python code on the fly. Even though the same abstractions might show up, the advantage is that they're expressed using a framework that allows us to test almost anything. This way, we get both: abstractions, but also maximum flexibility when the abstractions don't cut it. This is the same approach we've taken with the AccessibleNode work: create a very raw API in the browser and utility functions which abstract away common tasks, thus giving us maximum flexibility long-term. That said, your declarative solutions to my example use cases seem very reasonable. For reference, I'll provide links showing how we do this in Gecko for each of them.
Here are Gecko's table tests for UIA. Note the use of reusable functions to avoid lots of code duplication, which highlights your point above.
Here are Gecko's (currently extremely limited) tests for IA2 text.
Unlike IA2 and ATK, where the text interface is based on offsets, UIA has range objects. The client never sees offsets, though it can be implemented using offsets by the server. Instead, the client gets a range of interest; e.g. get the range for an object, for an entire document, etc. It then moves the range around with various methods. For example, there's an ExpandToEnclosingUnit method which expands (or shrinks) a range to the nearest line, word, etc. Here is a Gecko test for ExpandToEnclosingUnit. I think this in particular would be really difficult to express declaratively. Mac is similar, though you get handed opaque marker blobs rather than objects. Here's a Gecko test to ensure that the caret returns the correct line when it's at the start of the line.
Sounds good!
Note that I think we will eventually need tests which do a thing, wait for an event, verify the state after that event, do another thing, verify the state, etc. This is not just about testing the events themselves, but also waiting for actions we've taken to complete so the tests are reliable, since layout and accessibility is somewhat async from DOM. Here's a simple Gecko test which verifies that we fire the correct scrolling event for MSAA/IA2 when scrolling to a text fragment. |
||
| "Atspi" : [ | ||
| [ | ||
| "property", | ||
| "objectAttributes", | ||
| "contains", | ||
| "autocomplete:both" | ||
|
cookiecrook marked this conversation as resolved.
|
||
| ], | ||
| [ | ||
| "property", | ||
| "states", | ||
| "contains", | ||
| "STATE_SUPPORTS_AUTOCOMPLETION" | ||
| ] | ||
| ], | ||
| "IAccessible2" : [ | ||
| [ | ||
| "property", | ||
| "objectAttributes", | ||
| "contains", | ||
| "autocomplete:both" | ||
| ], | ||
| [ | ||
| "property", | ||
| "states", | ||
| "contains", | ||
| "IA2_STATE_SUPPORTS_AUTOCOMPLETION" | ||
| ] | ||
| ] | ||
| } | ||
| ); | ||
|
|
||
| AAMUtils.verifyAPI( | ||
| 'input[role=combobox][aria-autocomplete=inline]', | ||
| 'test2', | ||
| { | ||
| "Atspi" : [ | ||
| [ | ||
| "property", | ||
| "objectAttributes", | ||
| "contains", | ||
| "autocomplete:inline" | ||
| ], | ||
| [ | ||
| "property", | ||
| "states", | ||
| "contains", | ||
| "STATE_SUPPORTS_AUTOCOMPLETION" | ||
| ] | ||
| ], | ||
| "IAccessible2" : [ | ||
| [ | ||
| "property", | ||
| "objectAttributes", | ||
| "contains", | ||
| "autocomplete:inline" | ||
| ], | ||
| [ | ||
| "property", | ||
| "states", | ||
| "contains", | ||
| "IA2_STATE_SUPPORTS_AUTOCOMPLETION" | ||
| ] | ||
| ] | ||
| } | ||
| ); | ||
|
|
||
| AAMUtils.verifyAPI( | ||
| 'input[role=combobox][aria-autocomplete=list]', | ||
| 'test3', | ||
| { | ||
| "Atspi" : [ | ||
| [ | ||
| "property", | ||
| "objectAttributes", | ||
| "contains", | ||
| "autocomplete:list" | ||
| ], | ||
| [ | ||
| "property", | ||
| "states", | ||
| "contains", | ||
| "STATE_SUPPORTS_AUTOCOMPLETION" | ||
| ] | ||
| ], | ||
| "IAccessible2" : [ | ||
| [ | ||
| "property", | ||
| "objectAttributes", | ||
| "contains", | ||
| "autocomplete:list" | ||
| ], | ||
| [ | ||
| "property", | ||
| "states", | ||
| "contains", | ||
| "IA2_STATE_SUPPORTS_AUTOCOMPLETION" | ||
| ] | ||
| ] | ||
| } | ||
| ); | ||
|
|
||
| </script> | ||
|
|
||
| </body> | ||
| </html> | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,64 @@ | ||
| <!doctype html> | ||
| <meta charset=utf-8> | ||
| <html> | ||
| <head> | ||
| <title>aria-braillelabel</title> | ||
| <script src="/resources/testharness.js"></script> | ||
| <script src="/resources/testharnessreport.js"></script> | ||
| <script src="/resources/testdriver.js"></script> | ||
| <script src="/resources/testdriver-vendor.js"></script> | ||
| <script src="/resources/testdriver-actions.js"></script> | ||
| <script src="/core-aam/scripts/aam-utils.js"></script> | ||
| </head> | ||
| <body> | ||
|
|
||
| <p> | ||
| This test is marked tentative because it uses an experimental testing API | ||
| that is presently under development. | ||
| </p> | ||
|
|
||
| <button id=test aria-braillelabel=foobar> | ||
|
|
||
| <script> | ||
| AAMUtils.verifyAPI( | ||
| 'button[aria-braillelabel=foobar]', | ||
| 'test', | ||
| { | ||
| "Atspi" : [ | ||
| [ | ||
| "property", | ||
| "objectAttributes", | ||
| "contains", | ||
| "braillelabel:foobar" | ||
| ] | ||
| ], | ||
| "AXAPI" : [ | ||
| [ | ||
| "property", | ||
| "AXBrailleLabel", | ||
| "is", | ||
| "foobar" | ||
| ] | ||
| ], | ||
| "IAccessible2" : [ | ||
| [ | ||
| "property", | ||
| "objectAttributes", | ||
| "contains", | ||
| "braillelabel:foobar" | ||
| ] | ||
| ], | ||
| "UIA" : [ | ||
| [ | ||
| "property", | ||
| "AriaProperties.braillelabel", | ||
| "is", | ||
| "foobar" | ||
| ] | ||
| ] | ||
| }, | ||
| ); | ||
| </script> | ||
|
|
||
| </body> | ||
| </html> |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,78 @@ | ||
| <!doctype html> | ||
| <meta charset=utf-8> | ||
| <html> | ||
| <head> | ||
| <title>aria-errormessage</title> | ||
| <script src="/resources/testharness.js"></script> | ||
| <script src="/resources/testharnessreport.js"></script> | ||
| <script src="/resources/testdriver.js"></script> | ||
| <script src="/resources/testdriver-vendor.js"></script> | ||
| <script src="/resources/testdriver-actions.js"></script> | ||
| <script src="/core-aam/scripts/aam-utils.js"></script> | ||
| </head> | ||
| <body> | ||
|
|
||
| <p> | ||
| This test is marked tentative because it uses an experimental testing API | ||
| that is presently under development. | ||
| </p> | ||
|
|
||
| <div role='checkbox' id='test' aria-errormessage='error1 error2' aria-invalid='true'>content</div> | ||
| <div id='error1'>hello</div> | ||
| <div id='error2'>world</div> | ||
|
|
||
| <script> | ||
| AAMUtils.verifyAPI( | ||
| "div[role=checkbox][aria-errormessage='error1 error2'][aria-invalid='true']", | ||
| 'test', | ||
| { | ||
| "Atspi" : [ | ||
| [ | ||
| "relation", | ||
| "RELATION_ERROR_MESSAGE", | ||
| "is", | ||
| ['error1', 'error2'] | ||
| ], | ||
| [ | ||
| "reverseRelation", | ||
| "RELATION_ERROR_FOR", | ||
| "is", | ||
| ['error1', 'error2'] | ||
| ] | ||
| ], | ||
| "AXAPI" : [ | ||
| [ | ||
| "property", | ||
| "AXErrorMessageElements", | ||
| "is", | ||
| ['error1', 'error2'] | ||
| ] | ||
| ], | ||
| "IAccessible2" : [ | ||
| [ | ||
| "relation", | ||
| "IA2_RELATION_ERROR", | ||
| "is", | ||
| ['error1', 'error2'] | ||
| ], | ||
| [ | ||
| "reverseRelation", | ||
| "IA2_RELATION_ERROR_FOR", | ||
| "is", | ||
| ['error1', 'error2'] | ||
| ] | ||
| ], | ||
| "UIA" : [ | ||
| [ | ||
| "property", | ||
| "ControllerFor", | ||
| "is", | ||
| ['error1', 'error2'] | ||
| ] | ||
| ] | ||
| } | ||
| ); | ||
| </script> | ||
|
|
||
| </body> | ||
| </html> |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,82 @@ | ||
| <!doctype html> | ||
| <meta charset=utf-8> | ||
| <html> | ||
| <head> | ||
| <title>role blockquote</title> | ||
| <script src="/resources/testharness.js"></script> | ||
| <script src="/resources/testharnessreport.js"></script> | ||
| <script src="/resources/testdriver.js"></script> | ||
| <script src="/resources/testdriver-vendor.js"></script> | ||
| <script src="/resources/testdriver-actions.js"></script> | ||
| <script src="/core-aam/scripts/aam-utils.js"></script> | ||
| </head> | ||
| <body> | ||
|
|
||
| <p> | ||
| This test is marked tentative because it uses an experimental testing API | ||
| that is presently under development. | ||
| </p> | ||
|
|
||
| <div id=test role=blockquote>quote</div> | ||
|
|
||
| <script> | ||
| AAMUtils.verifyAPI( | ||
| 'div[role=blockquote]', | ||
| 'test', | ||
| { | ||
| "Atspi" : [ | ||
| [ | ||
| "property", | ||
| "role", | ||
| "is", | ||
| "block quote" | ||
| ] | ||
| ], | ||
| "AXAPI" : [ | ||
| [ | ||
| "property", | ||
| "AXRole", | ||
| "is", | ||
| "AXGroup" | ||
| ], | ||
| [ | ||
| "property", | ||
| "AXSubrole", | ||
| "is", | ||
| "<nil>" | ||
| ] | ||
| ], | ||
| "IAccessible2" : [ | ||
| [ | ||
| "property", | ||
| "role", | ||
| "is", | ||
| "IA2_ROLE_BLOCK_QUOTE" | ||
| ], | ||
| [ | ||
| "property", | ||
| "msaaRole", | ||
| "is", | ||
| "ROLE_SYSTEM_GROUPING" | ||
| ] | ||
| ], | ||
| "UIA" : [ | ||
| [ | ||
| "property", | ||
| "ControlType", | ||
| "is", | ||
| "Group" | ||
| ], | ||
| [ | ||
| "property", | ||
| "LocalizedControlType", | ||
| "is", | ||
| "blockquote" | ||
| ] | ||
| ] | ||
| } | ||
| ); | ||
| </script> | ||
|
|
||
| </body> | ||
| </html> |
Uh oh!
There was an error while loading. Please reload this page.