Skip to content

Android Results Capture Prototype Workflow

Jump to bottom
Howard Edwards edited this page Jul 24, 2025 · 9 revisions

Background

The current focus of the ARIA AT-App is the manual and automated testing of browsers and ATs rendering of ARIA accessibility features on desktop or laptop computers. One of the goals for the next year is to expand the testing of ATs renderings of ARIA accessibility features on mobile devices. This will include Android and iOS.

Today, Testers observe AT behavior and report it via ARIA-AT App--all using the same device. The introduction of a second physical device into the testing workflow presents the largest challenge for mobile support. Testers will need to collect AT responses from their mobile device and report that information using ARIA-AT App on their primary computer.

To facilitate this workflow, we are building the Android Results Capture Prototype. It will allow Testers to easily move text data from their Android device to their primary computer via the clipboards of each system.

This document serves as an instructional document on how to test the Android Results Capture Prototype and how to provide feedback.

Objective of the prototype testing

Through the testing of this prototype we are trying to learn:

  1. Is this approach practical for you as a tester, test authors, or administrator?
  2. What difficulties did you encounter while using the prototype for mobile testing?
  3. How would you improve the workflow?
  4. Could you envision yourself using this solution for a complete test run? If not, why not?
  5. Is there any reason we should not replicate this approach for iOS?

Please add your feedback about the prototype to this GitHub issue. We’ll use the feedback we receive about the Android testing prototype to finalize a solution for Android and to implement a solution for iOS.

Description of the Prototype

Changes to aria-at

For test authors and admins, this prototype adds:

  1. A collection of sample mobile gesture commands.
  2. TalkBack for Android specific metadata, assertion tokens and instructions that are specific to TalkBack for Android (in the support.json file, just like the ATs that are already supported).
  3. A sample talkback_android-commands.csv for the alert test plan.

This is viewable in the aria-at PR Preview here and inARIA-AT App here. You will notice a new TalkBack for Android filter button.

Changes to the aria-at-app

For testers and other users, this prototype modifies the Test Queue. Under the Alert Example h2, if a user expands the V25.04.29 disclosure they will now have a new row item for TalkBack for Android + Chrome for Android row item.

How to test the prototype

The prototype enables manual testers to collect screen reader utterances from mobile devices by importing the collection of utterances from their mobiles to the clipboard of their primary computer. To do this, testers need to first configure their Android device for testing, then physically connect their Android device to their computer, and finally access the prototype on the ARIA-AT App sandbox environment from their computer.

Below is a detailed guide for this testing process.

Step 1: Software check

Please check if you have the following software installed in your Android device. If you do not have the software, please use the associated links to download and install it.

Step 2: Enable Developer Options

  1. Open the Settings app on your Android device
  2. Scroll down and tap "About phone"
  3. Find "Build number" (On some devices, this may be under "Software information")
  4. Tap "Build number" 7 times quickly (You may be prompted to enter your PIN, password or pattern)
  5. A toast message will appear saying "You are now a developer!". If this was already done on the device, instead you will get a toast message which says "No need, you are already a developer."

Step 3: Turn On USB Debugging

  1. Go back to the main Settings screen.
  2. Scroll down and tap "System" (on some devices, you may see "Additional settings" or just "Developer options" here).
  3. Tap "Developer options".
  4. Some devices may have a "Use developer options" toggle at the top of this page. Make sure to enable that.
  5. Scroll down to the "Debugging" section.
  6. Toggle "USB debugging" on.
    • Note: Some devices may have “USB debugging (Security Settings)”. Toggle that option on as well if present.
  7. A confirmation dialog will appear. Tap "OK" to confirm.

Step 4: Connect to Your Computer

  1. Plug your device into your computer using a USB cable.
  2. You should see a prompt on your Android asking: "Allow USB debugging?"
    • This may not appear until running the local proxy described below in Step 5
  3. Tap "Allow" and optionally check "Always allow from this computer if you don’t want to be prompted again.

Step 5: Testing the new workflow in the app

  1. Navigate to the Test Queue.
  2. Under the Alert Example h2, expand the V25.04.29 disclosure and view the tests for TalkBack for Android + Chrome for Android.
  3. Download the local proxy zip for your operating system:
    1. Windows
    2. macOS
  4. Unzip and start the proxy using the included start script.
    • start.bat on Windows. (Note that you may have to allow permissions through Windows SmartScreen to continue on the first run by doing the following):
      1. Open start.bat
      2. On the pop-up which mentions "Windows protected your PC Microsoft Defender SmartScreen prevented an unrecognized app from starting. Running this app might put your PC at risk.", select "More Info"
      3. Select "Run Anyway"
      4. Another window titled "ADB Proxy for ARIA-AT Testing" may appear, requesting permission to open.
      5. Select "Allow"
    • start.command on macOS. (Note that you may have to allow permissions through macOS' Privacy & Security settings to continue on the first run by doing the following):
      1. Open start.command
      2. If a prompt appears notifying you that the program is from an unknown developer, open System Settings
      3. Go to Privacy & Security
      4. Scroll down to the Security section
      5. A message will be there which reads: "adb-proxy was blocked from use because it is now from an identified developer."
      6. Select "Open Anyway"
      7. Attempt to open it once again
  5. Click the 'Open Test Page on Android Device' button:
    • This would enable TalkBack on the Android device
    • This would open the example test page in Google Chrome on the Android device
    • This would immediately start a "listener" which waits for the "Run Test Setup" button on the example test page to be activated.
    • When the user activates the "Run Test Setup" button, as they move through the example with the instructed commands, the utterances from TalkBack are being captured through a web socket. When the user gets to the end of the example (which is also uttered) or the Android device's browser window is closed, the utterances would be copied to their connected computer's clipboard. The user can also copy the utterances directly with the "Copy utterances to clipboard" button from the 'Captured Utterances' section.
    • (Optional) Close the Android device's browser tab of the example page
  6. Paste the content currently on the connected computer's clipboard into the relevant output <textarea>

The following screenshot shows the TalkBack utterances being displayed in a debug view on the Test Run page. It also shows the messages being received through the WebSocket in the Browser's Console:

screenshot showing a snippet of captured TalkBack utterance on screen in Test Run page for alert example

Additional context

The deployed changes involves work from:

Previous discussions:

Clone this wiki locally