Repository navigation
android: expose GPUI's accessibility tree to TalkBack - #25
Open
Bombatomica64 wants to merge 5 commits into
Open
Bombatomica64 wants to merge 5 commits into
Bombatomica64 wants to merge 5 commits into
Conversation
GPUI 0.3.7 builds an AccessKit tree and hands it to PlatformWindow::a11y_init / a11y_tree_update, which the Android window left as no-ops. Wire them to accesskit_android's InjectingAdapter, which installs an AccessibilityDelegate on the existing host View, so hosts need no Java changes. accesskit_android 0.7.5 is the newest release on accesskit 0.24 (the version gpui-pre 0.3.7 uses). It raises accessibility events without checking AccessibilityManager.isEnabled(), and Android throws on the UI thread when no service is listening, so updates are skipped while accessibility is off (0.9 has this check but needs accesskit 0.25). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Only the sibling window and jni modules call it, matching document_picker and text_input. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
InjectingAdapter is already Send (its JavaVM, WeakRef, global JClass and Arc<Mutex<_>> fields all are in jni 0.21), so the bridge state can live in the static Mutex without it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Once accessibility turns off, tell GPUI to stop building trees and swap in an adapter that is not yet active. Before, the adapter stayed active after TalkBack was turned off, so turning it back on never re-activated GPUI and TalkBack read a stale tree until something repainted; GPUI also kept building trees nobody read. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
On the android-activity (NativeActivity) path every motion event goes to the native input queue first, and hover events were reported as handled even though GPUI skips them. With TalkBack on, touch exploration arrives as hover events, so the decor view's accessibility delegate never saw them and TalkBack found nothing under the finger. Report them unhandled so Android passes them on, as touches on platform views already are. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Author
|
I added two fixes after more testing, each as its own commit:
I found both while porting this to the current Zed GPUI for itsbalamurali/gpui-mobile, whose example only runs as a NativeActivity. Verified with TalkBack on a OnePlus CPH2581 (Android 16), and with repeated activate/deactivate cycles on redroid. |
Bombatomica64
added a commit
to Bombatomica64/gpui-mobile
that referenced
this pull request
Oct 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
GPUI 0.3.7 already builds an AccessKit tree and hands it to
PlatformWindow::a11y_init/a11y_tree_update, but the Android window leaves both as no-ops, so TalkBack only sees one opaque surface. This PR connects them toaccesskit_android'sInjectingAdapter. That adapter installs anAccessibilityDelegateon the existing hostView, so hosts don't need any Java changes.It's one new module (
src/android/accessibility.rs) plus two hooks inwindow.rsand two small changes injni.rs, about 220 lines in total.How it works
a11y_initstores GPUI's callbacks and attaches the adapter to the host view. That's the content view on the host-driven path, where theSurfaceViewlives, and the decor view on theandroid-activitypath.a11y_tree_updateforwards each tree to the adapter. The adapter posts its work to the UI thread itself.set_host_activitymoves the adapter onto the new Activity's views when the Activity is recreated.android-activitypath, hover events are now reported as unhandled. GPUI doesn't use hover on mobile, and TalkBack's touch exploration arrives as hover events, which the delegate on the decor view needs to see.Two things worth flagging
accesskit_android0.7.5 is the newest release built onaccesskit0.24, which is the versiongpui-pre0.3.7 uses. It can move to 0.9 when GPUI moves toaccesskit0.25.AccessibilityManager.isEnabled(). Android throws on the UI thread when no service is listening, so the app crashed as soon as the screen reader disconnected. This PR checks first. Once accessibility is off, it calls GPUI'sdeactivationcallback and attaches a fresh adapter, so turning the screen reader back on activates GPUI again instead of reading a stale tree. 0.9 does theisEnabled()check internally, so most of this can go away with the upgrade.Testing
Tested with the host-driven path in a small lab app on GPUI Kit 0.7.0.
OnePlus CPH2581 (Android 16), TalkBack. Touch exploration lands on the right controls, double-tap toggles a checkbox, and turning TalkBack off with the volume keys no longer crashes the app:
Redroid (Android 14),
uiautomator dump. Every node's bounds drawn over the rendering. Kit's checkboxes, radios, switches and sliders come through as their native Android classes with labels and checked state:android-activity(NativeActivity) path, same phone. I tested the same bridge ported to the NativeActivity example in itsbalamurali/gpui-mobile#69, because the lab app uses the host-driven path. That's where the hover fix above came from: before it, touch exploration never reached the delegate. With it, TalkBack reads the controls and double-tap toggles them. One limitation remains: Android doesn't draw its green focus ring, because NativeActivity takes over the window surface that the ring would be drawn on.Happy to change the approach if you'd prefer something different, for example a Java-side
AccessibilityNodeProviderin the host instead of the injected delegate.🤖 Generated with Claude Code