Accessibility Testing: A Green Scan Is Not a Finished Journey

Your accessibility dashboard is green. Your customer still cannot pay. Both things can be true, because a scanner measures rules it can inspect, not whether a person can finish what they came to do. The mistake is not using automation. The mistake is promoting one kind of evidence into a conclusion it cannot support.

Accessibility testing needs three complementary layers: automated checks, knowledgeable manual evaluation against the relevant standard, and research with people who use the product through different access methods. W3C explicitly says that no evaluation tool alone can determine whether a site meets accessibility standards. That is not an argument against tools. It is an argument for giving them the right job.

What automation knows—and what it does not

Automated tools are excellent at repeatable inspection: unnamed controls, some contrast failures, invalid relationships, structural problems that can be detected from the page. Run them frequently. Put them in development checks so yesterday's correction does not become tomorrow's regression. Their cheap repeatability is exactly the point.

But a meaningful alternative text is not the same as an existing alternative text. A dialog can have the right attributes and still strand keyboard focus. A form can announce an error while giving recovery instructions that nobody understands. A purchase can satisfy many page-level checks while its external payment step blocks completion. The task crosses the boundary of the scan.

Do not attach an invented universal percentage to what scanners catch. Detection depends on the tool, the page, the criteria, the configuration and what counts as a defect. The exhibit below uses six fictional barriers to make the distinction visible; it is deliberately not a market benchmark.

Conformance and usability answer different questions

WCAG conformance asks whether content meets specified success criteria at a specified level. Usability research asks whether people can achieve a goal in context, where they struggle, and what helps them recover. A usability session does not certify conformance. A conformance report does not establish that the whole experience is clear, efficient or respectful.

W3C's guidance on involving users makes that distinction useful rather than adversarial: disabled participants can reveal problems that a checklist does not capture. GOV.UK similarly recommends automated and manual testing alongside research with disabled and older users. Its Service Standard expects WCAG 2.2 AA and an audit before public beta; that is a requirement for those services, not a universal rule for every company.

In July 2026, W3C published WCAG-EM 2.0, a Group Note describing a structured evaluation method: define the scope, explore the product, choose a representative sample, evaluate, and report. It is guidance for organizing evidence, not a new level of WCAG compliance. The product lesson is powerful: define what you assessed before saying what passed.

The journey is your unit of quality

Start with the things users cannot afford to get wrong: sign in, book, buy, submit, recover, cancel, contact support. Write a realistic task with a beginning and an observable completion state. Include the negative path. A perfect checkout with an unusable payment-error screen is not a perfect checkout.

Take a hypothetical travel booking. The task is not “inspect the date picker.” It is “choose two dates, understand the fare, correct a rejected payment, and find the confirmation.” That formulation exposes dependencies the design review might miss: the calendar, account authentication, the embedded provider, the error message and the receipt. Assign an owner across those seams.

The same logic applies to an accessible AI chat interface. Test the reply, the Stop action, a failed request and escalation to a human—not merely whether the text field has a label. An assistant that is the only support channel creates a particularly consequential failure if people cannot operate it.

Recruit for access needs, not a checkbox

There is no single “disabled user” configuration. Screen readers, magnification, keyboard-only input, voice control, cognitive access needs and combinations of these create different interactions. Choose participants and tasks based on the audiences and risks of the service. Include people who already use their assistive technology confidently; a researcher learning it during the session is not equivalent evidence.

Ask about accessible session arrangements ahead of time. Let participants use their familiar equipment where practical. Provide consent information in a usable format, allow breaks, compensate their time and avoid unnecessary collection of medical details. The research should not impose the very access barriers it is meant to expose.

Do not prompt someone past the obstacle and then count the task as independent success. Record whether the person completed independently, with help, or not at all. Note the exact obstacle and environment. A small session can reveal a serious barrier; it cannot estimate the population prevalence of that barrier with precision.

Turn a finding into a release decision

A useful finding contains the task, expected behavior, observed barrier, affected interaction method, reproduction steps and completion consequence. “Accessibility issue” is too vague to prioritize. “Keyboard focus cannot reach the payment retry control after failure” gives the team a repair and a retest.

Prioritize completion blockers and safety-relevant misunderstandings before cosmetic cleanup. Fix the underlying pattern, then inspect every sibling that uses it. A focus repair in one dialog is not finished if six dialogs share the same behavior. Keep an automated regression where it can test the property, and preserve the manual task where automation cannot.

This is accessibility as a product decision, not a ceremony at the end of delivery. Product managers own the acceptance condition and the vendor boundary; engineering owns reliable behavior; research supplies observed human evidence. Nobody should be able to point at a green tool and outsource the conclusion.

A practical two-week evidence sprint

1. Define the slice. Pick one consequential journey, supported environments and the standard you are evaluating. State exclusions explicitly.
2. Inspect and walk. Run automated checks, then keyboard, zoom and assistive-technology evaluation with a knowledgeable tester. Preserve failures, not just totals.
3. Observe people. Rehearse the complete task with relevant participants, including recovery. Separate facilitation from independent completion.
4. Repair and retest. Fix shared patterns, repeat the failed task and record what remains. Publish a truthful statement rather than an absolute claim.

Two weeks is a proposed starting cadence, not a guaranteed audit duration. Complex services need more scope, expertise and time. If the European Accessibility Act reaches your offering, get qualified advice about the actual legal requirements as well.

The product outcome is a stronger definition of done. Not “no warnings.” Not “someone clicked through.” A named journey, independently completed, supported by criteria-based evaluation and evidence that another person can reproduce. That is how a team makes accessibility visible before a customer has to make the failure visible for them.

Sources and limits

Primary sources support the standards and policy descriptions. The planning tools and proposed release practices are editorial models, not audit certificates, legal advice or measured production results. Provider documentation can change; verify the applicable agreement before procurement.

• W3C: Accessibility Conformance Evaluation Methodology (WCAG-EM) 2.0, Group Note, July 23, 2026
• W3C WAI: Evaluating Web Accessibility
• W3C WAI: Involving Users in Evaluating Web Accessibility
• GOV.UK: Testing for accessibility