Accessibility APIs: Where the magic happens!

Hi there.

Thanks for having me.

Today I'm going to talk to you about accessibility APIs.

So I'd like to kick things off with a bit of a story.

I was working with a client a couple of years ago who had a form with a series of checkboxes in a two column layout.

We were testing the form and we noticed that when we got down to the check boxes, the group was being announced as a table by voiceover on iOS.

"Table start" at the beginning "table end" at the end.

We looked at the markup, which for the most part looked fine.

The checkboxes were associated to their labels using a for and ID connection.

And there wasn't a table in sight.

It turns out they were using display: table to achieve the two column layout.

And after plugging in the phone and inspecting the accessibility tree, we worked out that for whatever reason that caused the container to be rendered as a table by voiceover.

Didn't happen with JAWS, and it didn't happen within NVDA.

All from memory, with talk back on Android.

I'll dig into how the technology works, to hopefully give you a bit more of an understanding of where the disconnect happened and how to debug.

But before I get into any technical details, there's a few terms that I'll use consistently throughout this presentation, that I'd like to define.

The accessibility tree: accessibility tree is a subset of the DOM that is a hierarchical representation of the accessibility information of elements in the UI.

Accessibility APIs: accessibility APIs are a collection of interfaces to facilitate the communication of accessibility information about a user interface to assistive technologies.

They exist on all platforms, and as you'll find out a bit later on there's more than one and browsers can implement multiple.

Assistive technology: assistive technology is software and/or hardware that people with disability use to improve their interaction with the web and technology.

The accessibility tree accessibility APIs, assistive technologies, your markup and web browsers, all work together to make magic happen.

Let's take a bit of a peek behind the curtain to see how it all works.

It all starts with the markup because as we all know, the markup is the foundation of any website.

Every accessibility consultant ever will tell you to use semantic HTML and native HTML elements, whenever you can.

And that's because using the right elements for the right things will give you a lot of the accessibility you need for free.

I know Hidde will dive into this for his presentation.

So go check that out.

But in short semantic, HTML provides meaning and structure to web content.

If we take a look at this demo page that I've put together for this talk, I've used the following bits of semantic markup.

I've used landmarks and regions, I've used headings and I've used form controls like fieldsets, legends input fields, radio buttons, and buttons.

I haven't used any ARIA roles because for what I was doing, I didn't need them.

But if you needed to add semantics using ARIA, you could do that too.

All right.

So let's look at some markup.

Here we've got got a set of radio buttons, wrapped in a fieldset and labeled with a legend.

We're going to use this markup and these radio buttons, as our example for the rest of the talk.

The browser uses this HTML plus any extra relevant info it gets from the CSS and builds the accessibility tree.

And as we know, from our definition earlier, the accessibility tree is a hierarchical representation of the accessibility information of elements in the UI being displayed.

But what's in it?

Well for each element the accessibility tree holds objects, just like the DOM does.

So for controls, it would hold the name of the control, so it's label or the label, the role of the control.

So for example, is it a button or link?

The condition the control's in.

So is it checked, unchecked, expanded, or collapsed?

It's properties, any other characteristics ti might have.

So for example, is it required?

And any relationships that has like, is it tied to another element on the page or is it part of a group?

The accessibility tree should have everything assistive technology needs to inform users about what's on the page.

Let's walk through an example element by element, using the markup for the radio buttons we saw earlier.

I'll simplify property names and values, because they can vary between accessibility, APIs, and it'll just make things easier to explain.

So first up is our fieldset element.

The fieldset has a legend that will give it its accessible name and the HTML mapping standards specify that a fieldset would be mapped as a group.

Because the fieldset is given a name by its legend, the fieldset's object properties will also store a labeled-by relationship pointing to that element.

If we took a look at our tree we'd see a role of 'group' for our field set.

Next up, we've got the legend.

This doesn't have an accessible name.

It's just used as text to label the fieldset.

If we look at our updated tree though, we'd see the fieldset group is now labeled by the legend and has a name of "which team won the 2021 AFL grandfinal".

Now let's add the radio button input, which we mapped as a radio role.

I'm going to add the label here too, because it makes sense.

And we saw the concept of labeling with the legend earlier.

Because the 'for' attribute of the label is pointing to the ID attribute of the Input, the radio button will take its accessible name from the label text.

The radio buttons object properties will also store the labeled-by relationship pointing to that label.

And as we can see from this code, this particular radio button is checked using a check attribute, so we'll store that in the object property as a state as well.

Looking at the updated tree, we'd see our radio button is nested within the field set group with the name of Melbourne, a checked property of true and a labeled-by property pointing to its label to establish the relationship.

Finally, let's add the second radio button and complete our tree.

Like the previous radio input, it'll be mapped as a radio role, will take its name from its label and store the label by relationship pointing to the relevant label element.

Unlike the first radio button though, this one's not checked, so unchecked will be stored as the state in the object properties.

Our tree build for this set of radio buttons is now complete.

Both radio buttons are nested within the fieldset group, they're getting they're accessible names.

They're labeled-by properties are pointing to that labels and their states are checked and unchecked.

It's also worth pointing out that not all elements are included in the accessibility tree.

There was some rules around when browsers should include or exclude elements.

One of those rules is that elements are excluded from the accessibility tree, for example, if they hidden using CSS display: none or visibility: hidden, if they're hidden using the HTML 'hidden' attribute or aria-hidden = true, or if they have a role of 'presentation' or a role of 'none'.

Elements are also excluded from the accessibility tree if their children have particular roles.

The specifics around these rules and how they work are located in the accessibility tree section of the ARIA specifications.

So if you're interested, I'd suggest taking a look at that.

When we were building our accessibility tree earlier, I mentioned that the fieldset got its accessible name from the legend and that the radio buttons got their accessible names from the labels.

But the way an accessible name gets calculated can sometimes be confusing.

So it's worth a quick walkthrough.

An element's accessible name can come from many sources.

It can come from an element's content.

So link, text, or button text.

It can come from an attribute such as an alt attribute for an image.

It can come from an associated element.

So a radio, a label for a radio button, or a legend for a fieldset.

And it can come from ARIA attributes like aria-label and aria-labeledby.

Quick note, though, if you're going to use aria-label and aria-labeledby on an element, they take precedence in calculating the accessible name.

Even if the element's got content within it that should name it.

The accessibility APIs are the bridge between the browser, the accessibility tree, and assistive technologies.

Kind of like translators, assistive technologies use these accessibility APIs to interact with the accessibility tree, web content and any other information that's exposed by the browser.

As I mentioned previously, the different platforms and different operating systems each have different accessibility APIs.

Windows has MSAA with IAccessible2 and UI automation.

Mac OS has NSAccessibility.

Linux has Accessibility Toolkit and Assistive Technology Service Provider Interface.

iOS has UIAccessibility and Android has AccessibilityNodeInfo, and AccessibilityNodeProvider.

Browsers can implement more than one accessibility API.

So for example, a browser on windows can choose to support MSAA with IAccessible2 or UIAutomation or both and browsers that are available across platforms will implement multiple accessibility APIs.

So up to this point, we've talked about how the accessibility tree gets built by the browser based on the markup in a website.

But what happens when something changes?

Let's take a look at a quick example.

[Screen Reader speaks] Upload button.

To activate press enter.

Enter 0% uploading 24%.

Uploading 49% uploading 74%.

Upload complete.

The assistive technology is listening for events.

If content changes or a user performs an action, like uploading a file, the browser sends the accessibility API a notification about what's happened and the API passes this notification onto the assistive technology.

So in the example we just saw the screen reader is listening for events like a change to a live region.

The live region changes when the user presses the upload button.

The change event is triggered in the browser.

And a notification is sent to the accessibility API.

The API sends the notification to the screenreader and the screenreader uses the accessibility API to find the relevant object in the accessibility tree and announce the change to the user.

Okay.

So, so far we've talked about markup.

We've talked about how the accessibility tree gets built.

What's included and what's not included and how it all gets communicated from the browser to the assistive technology, through the accessibility API.

But what I haven't told you yet, it's possible to view the accessibility tree, right from the dev tools in your browser.

And this can come in quite handy when you're trying to work out why something isn't working the way it should.

That's right.

The dev tools in all modern browsers allow you to go back to the accessibility tree and see the information that will be sent to the accessibility API and sent on to assistive technologies.

Let's give it a go using Chrome dev tools and our radio buttons from our demo site earlier.

So first we're going to inspect the element using the dev tools.

We're going to bring up the accessibility tree for the page, selecting the accessibility tab in the right panel.

The accessibility tree accordion should be expanded, showing our radio buttons in the nested tree structure.

And if we expand the computer properties accordion, we should see at the same information that we saw earlier while building our tree.

With things like role, labeled-by and checked.

Different platforms will have different API viewers, just like they'll have different accessibility APIs.

But there'll also be some common ones as well.

If we were to look at Windows and Mac, as examples.

On Windows, you've got accessibility insights and the dev tools, in Edge, Chrome and Firefox.

And on Mac, we've got accessibility, inspector for OSX, and again, dev tools in browsers like Safari, Firefox and Chrome.

With iOS and Android, you can plug in your device and use the dev tools in Safari and Chrome to view the API or view the tree on mobile as well.

I told you a story at the beginning of this presentation, about an issue I had while working with a client.

The markup looked right, but the screen reader was incorrectly announcing something as a table because of the CSS display: table properly being applied.

At the time we used the accessibility API viewer to work out that that particular issue was one with the Safari browser on iOS setting display: table was sending the wrong information to the accessibility tree.

The API was then, as a result, sending the wrong information to the screen reader.

Let's take a look at a similar example, this time using display: flex and comparing Safari browser and Google Chrome on Mac OS.

[screen reader says] Heading level two, AFL grand final winners.

You are currently on a heading level two.

Table not found.

Table, not found.

Year.

Team 2021 Melbourne 2020 Richmond.

Heading level two, AFL grand final winnders.

You are currently on a heading level two.

Table, two columns, three rows.

You are currently in a table to navigate the cells within this table, press control option, and then up arrow, down arrow, left arrow or right arrow.

Year.

You are currently on a text element.

Row two of three.

2021.

Team Melbourne column 2 of 2.

[Adem] As we saw in that example, when we looked at the accessibility tree and Safari, display: flex was for some reason, removing the semantics from the table.

When we compared this with Chrome, we didn't have this problem.

And the table semantics were being communicated as we expected.

This tells us that in this case, the problem was something to do with the browser.

In this case Safari.

And the way it's building the accessibility tree.

In prepping for this talk.

I did check and this bug has been filed.

So hopefully it'll be fixed soon.

And that's it for me.

Thank you very much for having me.