Automating the accessibility tests of your source code with GitHub Actions

GitHub Actions

GitHub Actions allow you to automate, customize, and execute your software development workflows right in your repository with GitHub Actions. With GitHub Actions you could execute a series of statements and commands after a specific event has occurred, simply put, it's your own pipeline CI / CD directly in your repository.

Workflows

Github defines a workflow as a configurable automated process made up of one or more jobs. The workflow configuration to define events, tasks, and steps to be executed in GitHub Actions are defined using YAML files that must be located in the .github/workflows folder.

Using GitHub Actions to automate accessibility tests

Since we have seen what GitHub Actions are, let's see how we can use them to test the accessibility of your source code hosted on GitHub.

I have created a sample application in React with a small image component and some accessibility bugs. You can see the code in the project repository in my Github profile.

The first thing is to define when we want the GitHub Action to run. We can configure it to run immediately after each push to any of our branches (including the main branch).

name: example
on: [push]
jobs: ...

Or we can configure it to run on any Pull Request to our main main branch.****

name: example
on:
  push:
    branches: [main]
  pull_request:
    branches: [main]
jobs: ...

In my example repository I have decided to configure all Github Actions to run on any Pull Request to the main branch. Once we have decided when to execute the GitHub Action, we have to establish which events, tasks, and steps to execute.

Unit tests

In my first GitHub Action I want to run my unit tests when I create or update a Pull Request. Writing your own unit tests is the best way to find bugs and vulnerabilities in your code, and if you use jest, you can create your own accessibility unit tests with jest-axe, a tool from the axe family of tools.

$ npm install jest-axe --save-dev

import React from "react"
import App from "./App"
import { render } from "@testing-library/react"
import { axe, toHaveNoViolations } from "jest-axe"

expect.extend(toHaveNoViolations)

it("should not have any accessibility violations", async () => {
  const { container } = render(<App />)
  const results = await axe(container)
  expect(results).toHaveNoViolations()
})

Once the unit tests are created, I can now create my workflow in my GitHub Action. Luckily, GitHub offers you a bunch of templates when creating a new GitHub Action, and I've used the Node.js workflow template, to which I have deleted some comments and reduced the use of Node.js version to only use 12.x, which I will later use in all my GitHub Actions. This is what the final version of my workflow would look like:

name: unit-tests
on:
  push:
    branches: [main]
  pull_request:
    branches: [main]
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Use Node.js 12.x
        uses: actions/setup-node@v1
        with:
          node-version: 12.x
      - run: npm ci
      - run: npm run build --if-present
      - run: npm test

Now I just have to try it. After my Pull Request is created, my GitHub Action will start executing and the result will appear directly at the end of my Pull Request. You can see it in this Pull Request.

We will be able to access the details of the GitHub Action and see the results of the unit tests, to be able to solve the accessibility vulnerabilities in the code.

axe

axe is a family of tools created by Deque, which includes a command line interface (CLI), @axe-core/cli, which runs the axe search engine for accessibility vulnerabilities, and that we can use from a terminal. In my next GitHub Action I want to run that CLI on every Pull Request.

Keep in mind that @axe-core/cli is an informative tool, and that it only executes accessibility tests and displays the results on the screen. To make the execution of these tests cause an error in the execution we must add the option --exit to the axe command.

This is what the final version of my workflow would look like:

name: axe
on:
  push:
    branches: [main]
  pull_request:
    branches: [main]
jobs:
  axe:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Use Node.js 12.x
        uses: actions/setup-node@v1
        with:
          node-version: 12.x
      - run: npm ci
      - run: npm run build --if-present
      - run: npm start & npx wait-on http://localhost:3000
      - name: Run axe
        run: |
          npm install -g @axe-core/cli
          axe http://localhost:3000 --exit

As we have seen in the previous GitHub Action, when I go to the Pull Request that I had created and I have updated with a new commit, my two GitHub Actions will be executed again. You can see it in this Pull Request.

And we can always inspect the details of each GitHub Action to find out the accessibility vulnerabilities that caused the execution to fail.

axe-linter

GitHub also offers a library of applications, which we can install directly in our code repository, and which allow us to execute various tasks and events similar to GitHub Actions. One of them is axe-linter and you can find it directly on GitHub marketplace. You simply have to install it in your repository for free, and it will be ready to be used. This app, like my GitHub Actions, will run on every Pull Request and look for accessibility vulnerabilities.

The difference that I have seen compared to my own GitHub Action is the type of vulnerabilities it finds, as axe-linter can only find those vulnerabilities in HTML code written by you, and not in the HTML generated by my React application like what does @axe-core/cli. Anyway, I think it's a super useful app for determining if your code is accessible, so I'll keep it in my sample repository so you can see how it works. As with the GitHub Actions, and once I update my Pull Request (in this case I have added an explicit HTML error in my code), it will run alongside the other GitHub Actions. You can see it in this Pull Request.

As always, we can see the results of the tests, this time grouped by vulnerability.

pa11y

Like axe, there are other tools and applications that are very similar and equally valid for web accessibility testing. Another one is pa11y, a command line interface (CLI) tool that you can use from your terminal. In this case, unlike with axe, it is not just an informational tool and will cause an error in execution if it finds vulnerabilities in your code. Using my previous workflow as a base, this is what the final version would look like:

name: pa11y
on:
  push:
    branches: [main]
  pull_request:
    branches: [main]
jobs:
  pa11y:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Use Node.js 12.x
        uses: actions/setup-node@v1
        with:
          node-version: 12.x
      - run: npm ci
      - run: npm run build --if-present
      - run: npm start & npx wait-on http://localhost:3000
      - name: Run pa11y
        run: |
          npm install -g pa11y
          pa11y http://localhost:3000

As we've seen before, on every update to my Pull Request, all my GitHub Actions will be executed, including my axe-linter app as well. You can see it in this Pull Request.

And as in the previous ones, you can access the details of the test directly from the Pull Request.

Blocking the merge of a Pull Request if it has accessibility vulnerabilities

After creating all my GitHub Actions, and on each creation or update of any Pull Request, my code will be tested for accessibility vulnerabilities, but all those tests are informative at the moment, and I still have the final option to merge my Pull Request, which we don't want to happen.

To disable the button to merge any Pull Request that has accessibility vulnerabilities you will have to create a new branch protection rule in your repository. Access the Settings menu on the upper tab of your repository and then access Branches in the left menu. You should put an asterisk * in the field Branch name pattern and activate the checkbox Require status checks to pass before merging. You only have to save the changes by pressing the Save Changes button.

If you go back to your Pull Request, you will see that the Merge pull request button is disabled and merge cannot be done until the accessibility vulnerabilities are resolved and all GitHub Actions have satisfactory results. This way your application will be protected from accepting any inaccessible code.

Additional note: If you are the owner of the repository, you can check that you can always merge the Pull Requests. The protection will be effective for contributors.

Using GitHub Actions to automate reports with Lighthouse

We've seen how to create GitHub Actions that cause errors if they find accessibility vulnerabilities in your code, but I'd like to show you a GitHub Action for generating reports. Lighthouse is a tool created by Google, and it's included in the Google Chrome browser development tools, but it can also be run from the terminal. This tool generates reports on performance, accessibility, progressive web apps, SEO and more.

Using the lighthouse workflow template I have created a new GitHub Action to generate reports on each Pull Request.

Lighthouse

name: Lighthouse
on:
  push:
    branches: [main]
  pull_request:
    branches: [main]
jobs:
  lhci:
    name: Lighthouse
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Use Node.js 12.x
        uses: actions/setup-node@v1
        with:
          node-version: 12.x
      - name: npm install, build
        run: |
          npm install
          npm run build
      - name: run Lighthouse CI
        run: |
          npm install -g @lhci/cli@0.7.x
          lhci autorun

Lighthouse generates HTML reports, and you need to host them on a server. If your repository is public, you can configure it to host it on a temporary public storage. You will have to create a new file in the root of your project called lighthouserc.js with the following content. If you want to host the reports privately, you can read how to do it in the official Lighthouse documentation.

module.exports = {
  ci: {
    upload: {
      target: "temporary-public-storage",
    },
  },
}

Once my Pull Request is updated, it will run alongside the other GitHub Actions. You can see it in this Pull Request.

You will see that Lighthouse has a green tick, and not a red cross, as it is an informational tool and will not crash if your code has errors. If you inspect the execution details of the GitHub Action, and having configured the hosting of the reports publicly, a link to the generated report appears.

You simply need to access that report and use it to generate statistics, archive it to see the evolution of your website, or to obtain more information on possible vulnerabilities in accessibility, performance, progressive web apps, SEO and more.

Conclusion

After all these steps, my repository is configured, so my GitHub Actions are executed when I create or update any Pull Request to my main branch, including the axe-linter application. If any of the GitHub Actions find an accessibility vulnerability, my Pull Request would crash and disabled the merge option, so I would have to go through my code to resolve those errors. This way my application will be protected from accepting any non-accessible code.

You can see the code in the project repository in my Github profile, and the execution and details of the Github Actions in this Pull Request.

I recommend you to try creating your own GitHub Actions or use the ones I created if you find them useful.

If you want to continue learning about web accessibility, you can follow me on my twitter account, and do not hesitate to contact me or tweet me if you have any questions or doubt.