alphagov / accessibility-tool-audit

Automated accessibility tools audit

Accessibility tool audit

Automated accessibility checkers can be used to help identify accessibility issues in digital services. They're good for finding simple and obvious problems, but aren't able to detect many accessibility issues.

This repo contains a collection of accessibility failures to be used for testing automated accessibility tools and test results from those tools.

Read our blog post about how we did the automated tool testing.

About the test cases

The test cases are a collection of the wide variety of potential accessibility issues that can exist. There's probably loads more we haven't thought of.

Contributing / updating results

We welcome issues / pull requests for updated or new test cases or tool results. All relevant content can be found in tests.json. (All the HTML files are automatically created from that one file.) Read more on how to contribute.

Installing

Make sure you have the gulp command installed beforehand.

npm install -g gulp

Then run the following commands to install the dependencies and generate the static html files.

npm install
gulp

Now you can run a local HTTP server to serve the files in this directory. Such as:

python -mSimpleHTTPServer

and click http://localhost:8000/ to see the generated HTML output.

You can also run gulp in dev mode, which would keep watching for files to change until you kill it.

gulp dev

Licence

Released under the MIT Licence, a copy of which can be found in the file LICENCE.

accessibility-tool-audit's Issues

wmode only applies to Flash content

https://alphagov.github.io/accessibility-tool-audit/tests/html-object-not-embedded-accessibly-wmode-parameter-not-set-to-window.html

This test is ambiguous because wmode=transparent is only an issue if data="foo" refers to an SWF movie, because wmode is a proprietary Adobe parameter for Flash player.

Here's Adobe's wmode documentation:
https://helpx.adobe.com/flash/kb/flash-object-embed-tag-attributes.html#main_Browser_support_for_Window_Mode__wmode__values

So this is definitely an issue:

<object type="application/x-shockwave-flash" data="movie_name.swf" width="550" height="400">
        <param name="wmode" value="transparent" />
        Alternative text
</object>

this is definitely not an issue:

<!-- not Flash content: wmode has no effect -->
<object data="clock.html" type="text/html">
        <param name="wmode" value="transparent" />
        Alternative text
</object>

and this depends on whether data="foo" is an SWF file:

<!-- wmode has no effect unless "foo" is an SWF file -->
<object data="foo">
        <param name="wmode" value="transparent" />
        Alternative text
</object>

Is this project still active

Dear maintainers,

I was wondering if this project is still maintained ? I am willing to make a contribution to update the audit but would like to know if the project is still maintained before I do so.

Looking forward to hear from you,

Kind regards,

Diede

Should different img alt/title be an issue?

I'm not clear on why "Image has alt and title that are different" should be flagged by checkers.

According to the spec, when alt and title are both present, alt should be used as the accessible name, and title as the accessible description.

I believe SRs aren't compliant with this and tend to just ignore the title on img elements altogether, but the point here is that alt and title should not be the same, no?

"Large text" in colour contrast cases is not sufficiently large

The "large text" in the following two colour contrast test cases...

• https://alphagov.github.io/accessibility-tool-audit/tests/colour-and-contrast-large-text-does-not-have-a-contrast-ratio-of-at-least-45-1-so-does-not-meet-aaa.html
• https://alphagov.github.io/accessibility-tool-audit/tests/colour-and-contrast-large-text-does-not-have-a-contrast-ratio-of-at-least-31-so-does-not-meet-aa.html

...is not sufficiently large to be considered "large scale" from a WCAG perspective: https://www.w3.org/TR/WCAG/#dfn-large-scale. The WCAG definition of "large scale" requires that text is at least 18 point or 14 point bold; be observant of the units here, because point ≠ pixel. Per the CSS Values and Units Module specification (https://drafts.csswg.org/css-values/) 1 pixel is equivalent to 0.75 points, meaning that the font size of 20 pixels used for the "large text" is only 15 points.

Incorrect WAVE results recorded

The following test case results are incorrectly coded as "Not found" for WAVE (my comments in parentheses):

• Large text does not have a contrast ratio of at least 3:1 so does not meet AA (you must select the Contrast tab in WAVE to see these identified)
• Large text does not have a contrast ratio of at least 4.5:1 so does not meet AAA (you must select the Contrast tab in WAVE to see these identified)
• html element is missing a lang attribute (identified at top of the page)
• Missing page title (identified at top of the page)
• Missing H1 (identified at top of the page)
• Table with column headers and double row headers (WAVE alerts to the presence of the headers)
• Links to a sound file, no transcript (WAVE identifies the media link with documentation to verify transcript)
• Empty label found

All of these cases are clearly identified by WAVE and should be corrected in the results and the summary descriptions/blog post.

[Request] Include source/link/cite on Test-Cases.html

Would it be possible to add some reference/source material along with the things tested. I'd love to be able to reference the criteria that is behind the individual test. (see attached)

I realize WCAG is not the only standard used to determine accessibility - so even something simple like tags would be incredibly helpful.

Some test case results are incorrectly coded as "Not found" for Asqatasun

The following test case results are incorrectly coded as "Not found" for Asqatasun v4.0.3

"Link contains only a full stop"

https://alphagov.github.io/accessibility-tool-audit/tests/links-link-contains-only-a-full-stop.html

"html element has an empty lang attribute"
"Empty page title"

https://alphagov.github.io/accessibility-tool-audit/example-pages/empty.html

Seems like aXe catches many more problems than listed

Using the aXe browser plugin, many more of errors are detected that you did not list. I did not run an automated test, but it uses the same core...

Examples: html lang element, empty title...

href='#' isn't an invalid hypertext reference

https://alphagov.github.io/accessibility-tool-audit/test-cases.html#links-link-to-,-invalid-hypertext-reference

The behaviour is specified in the HTML 5 spec - it's a synonym for #top - both link to the top of the page. See step 2 and step 7 here:
https://www.w3.org/TR/html52/single-page.html#an-indicated-part-of-the-document

Also documented here:
https://developer.mozilla.org/en-US/docs/Web/HTML/Element/a#attr-href

Clarification on key statistics

Hello! I've been reading through the results and the awesome summary article, and I'm looking to verify my understanding of the key statistics presented. Apologies for not posting this elsewhere - it looks like comments are disabled on the blog, and the Google Group is private. Just let me know if there is another preferred avenue for questions like this :)

The biggest data point that I'm interested in understanding better is the total percentage of barriers that were not caught by the tools tested. There are two complementary statements that are made in the article that state that the tools tested found 71% of barriers in the test page, and missed 29% of them.

Under the heading "Lots of the barriers weren't found ...":

[...] a large proportion of the barriers we created weren’t picked up by any of the 10 tools we tested – 29% in fact.

Under the final heading, "How best to use automated tools":

[...] the tools picked up the majority of the accessibility barriers we created – 71% – [...]

When I look into the detailed audit results page, and the How did each tool do? section, I see that the best-performing tools found 40% of issues in the test setup.

I have a feeling that I'm missing a nuanced difference between the two numbers. Could someone provide some guidance on what those two different numbers (71% vs 40%) are referring to? They seem like a contradiction, which makes me think I may be misunderstanding something.

Run command has changed with Python 3

If people have Python 3 as their default Python, the command to run the web server is:
python3 -m http.server 8000

Security issues for Gulp and Minimatch

npm audit

                       === npm audit security report ===

# Run  npm install --save-dev [email protected]  to resolve 5 vulnerabilities
SEMVER WARNING: Recommended action is a potentially breaking change
┌───────────────┬──────────────────────────────────────────────────────────────┐
│ High          │ Regular Expression Denial of Service                         │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Package       │ minimatch                                                    │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Dependency of │ gulp [dev]                                                   │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Path          │ gulp > vinyl-fs > glob-stream > glob > minimatch             │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ More info     │ https://nodesecurity.io/advisories/118                       │
└───────────────┴──────────────────────────────────────────────────────────────┘


┌───────────────┬──────────────────────────────────────────────────────────────┐
│ High          │ Regular Expression Denial of Service                         │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Package       │ minimatch                                                    │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Dependency of │ gulp [dev]                                                   │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Path          │ gulp > vinyl-fs > glob-stream > minimatch                    │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ More info     │ https://nodesecurity.io/advisories/118                       │
└───────────────┴──────────────────────────────────────────────────────────────┘


┌───────────────┬──────────────────────────────────────────────────────────────┐
│ High          │ Regular Expression Denial of Service                         │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Package       │ minimatch                                                    │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Dependency of │ gulp [dev]                                                   │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Path          │ gulp > vinyl-fs > glob-watcher > gaze > globule > glob >     │
│               │ minimatch                                                    │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ More info     │ https://nodesecurity.io/advisories/118                       │
└───────────────┴──────────────────────────────────────────────────────────────┘


┌───────────────┬──────────────────────────────────────────────────────────────┐
│ High          │ Regular Expression Denial of Service                         │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Package       │ minimatch                                                    │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Dependency of │ gulp [dev]                                                   │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Path          │ gulp > vinyl-fs > glob-watcher > gaze > globule > minimatch  │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ More info     │ https://nodesecurity.io/advisories/118                       │
└───────────────┴──────────────────────────────────────────────────────────────┘


┌───────────────┬──────────────────────────────────────────────────────────────┐
│ Low           │ Prototype Pollution                                          │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Package       │ lodash                                                       │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Dependency of │ gulp [dev]                                                   │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Path          │ gulp > vinyl-fs > glob-watcher > gaze > globule > lodash     │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ More info     │ https://nodesecurity.io/advisories/577                       │
└───────────────┴──────────────────────────────────────────────────────────────┘


# Run  npm update minimatch --depth 5  to resolve 1 vulnerability
┌───────────────┬──────────────────────────────────────────────────────────────┐
│ High          │ Regular Expression Denial of Service                         │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Package       │ minimatch                                                    │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Dependency of │ gulp-sass [dev]                                              │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Path          │ gulp-sass > node-sass > true-case-path > glob > minimatch    │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ More info     │ https://nodesecurity.io/advisories/118                       │
└───────────────┴──────────────────────────────────────────────────────────────┘


found 6 vulnerabilities (1 low, 5 high) in 1937 scanned packages
  run `npm audit fix` to fix 1 of them.
  5 vulnerabilities require semver-major dependency updates.