Skip to content

android: expose GPUI's accessibility tree to TalkBack - #25

Open
Bombatomica64 wants to merge 5 commits into
longbridge:mainfrom
Bombatomica64:android-accessibility
Open

Bombatomica64 wants to merge 5 commits into
longbridge:mainfrom
Bombatomica64:android-accessibility

Conversation

@Bombatomica64

@Bombatomica64 Bombatomica64 commented Oct 5, 2026 •

Copy link
Copy Markdown

GPUI 0.3.7 already builds an AccessKit tree and hands it to PlatformWindow::a11y_init / a11y_tree_update, but the Android window leaves both as no-ops, so TalkBack only sees one opaque surface. This PR connects them to accesskit_android's InjectingAdapter. That adapter installs an AccessibilityDelegate on the existing host View, so hosts don't need any Java changes.

It's one new module (src/android/accessibility.rs) plus two hooks in window.rs and two small changes in jni.rs, about 220 lines in total.

How it works

  • a11y_init stores GPUI's callbacks and attaches the adapter to the host view. That's the content view on the host-driven path, where the SurfaceView lives, and the decor view on the android-activity path.
  • a11y_tree_update forwards each tree to the adapter. The adapter posts its work to the UI thread itself.
  • set_host_activity moves the adapter onto the new Activity's views when the Activity is recreated.
  • On the android-activity path, hover events are now reported as unhandled. GPUI doesn't use hover on mobile, and TalkBack's touch exploration arrives as hover events, which the delegate on the decor view needs to see.

Two things worth flagging

  • Version. accesskit_android 0.7.5 is the newest release built on accesskit 0.24, which is the version gpui-pre 0.3.7 uses. It can move to 0.9 when GPUI moves to accesskit 0.25.
  • Workaround. 0.7.5 raises accessibility events without checking AccessibilityManager.isEnabled(). Android throws on the UI thread when no service is listening, so the app crashed as soon as the screen reader disconnected. This PR checks first. Once accessibility is off, it calls GPUI's deactivation callback and attaches a fresh adapter, so turning the screen reader back on activates GPUI again instead of reading a stale tree. 0.9 does the isEnabled() check internally, so most of this can go away with the upgrade.

Testing

Tested with the host-driven path in a small lab app on GPUI Kit 0.7.0.

OnePlus CPH2581 (Android 16), TalkBack. Touch exploration lands on the right controls, double-tap toggles a checkbox, and turning TalkBack off with the volume keys no longer crashes the app:

Redroid (Android 14), uiautomator dump. Every node's bounds drawn over the rendering. Kit's checkboxes, radios, switches and sliders come through as their native Android classes with labels and checked state:

android-activity (NativeActivity) path, same phone. I tested the same bridge ported to the NativeActivity example in itsbalamurali/gpui-mobile#69, because the lab app uses the host-driven path. That's where the hover fix above came from: before it, touch exploration never reached the delegate. With it, TalkBack reads the controls and double-tap toggles them. One limitation remains: Android doesn't draw its green focus ring, because NativeActivity takes over the window surface that the ring would be drawn on.

Happy to change the approach if you'd prefer something different, for example a Java-side AccessibilityNodeProvider in the host instead of the injected delegate.

🤖 Generated with Claude Code

Bombatomica64 and others added 5 commits October 5, 2026 08:20
GPUI 0.3.7 builds an AccessKit tree and hands it to
PlatformWindow::a11y_init / a11y_tree_update, which the Android window
left as no-ops. Wire them to accesskit_android's InjectingAdapter, which
installs an AccessibilityDelegate on the existing host View, so hosts need
no Java changes.

accesskit_android 0.7.5 is the newest release on accesskit 0.24 (the
version gpui-pre 0.3.7 uses). It raises accessibility events without
checking AccessibilityManager.isEnabled(), and Android throws on the UI
thread when no service is listening, so updates are skipped while
accessibility is off (0.9 has this check but needs accesskit 0.25).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Only the sibling window and jni modules call it, matching document_picker
and text_input.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
InjectingAdapter is already Send (its JavaVM, WeakRef, global JClass and
Arc<Mutex<_>> fields all are in jni 0.21), so the bridge state can live in
the static Mutex without it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Once accessibility turns off, tell GPUI to stop building trees and swap in
an adapter that is not yet active. Before, the adapter stayed active after
TalkBack was turned off, so turning it back on never re-activated GPUI and
TalkBack read a stale tree until something repainted; GPUI also kept
building trees nobody read.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
On the android-activity (NativeActivity) path every motion event goes to
the native input queue first, and hover events were reported as handled
even though GPUI skips them. With TalkBack on, touch exploration arrives
as hover events, so the decor view's accessibility delegate never saw
them and TalkBack found nothing under the finger. Report them unhandled
so Android passes them on, as touches on platform views already are.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@Bombatomica64

Copy link
Copy Markdown
Author

I added two fixes after more testing, each as its own commit:

  • d4ae7c7. TalkBack turned off and back on read a stale screen. The adapter stayed active, so GPUI was never activated again, and it kept building trees nobody read. Now, once AccessibilityManager.isEnabled() is false, the bridge calls GPUI's deactivation callback and swaps in a fresh adapter. The next screen reader then activates GPUI cleanly.
  • a37bff0. Touch exploration on the android-activity (NativeActivity) path. Hover events went to the native input queue and were reported as handled even though GPUI skips them, so they never reached the decor view's accessibility delegate. They're now reported as unhandled, the same way touches on platform views already are.

I found both while porting this to the current Zed GPUI for itsbalamurali/gpui-mobile, whose example only runs as a NativeActivity. Verified with TalkBack on a OnePlus CPH2581 (Android 16), and with repeated activate/deactivate cycles on redroid.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant