Assistive virtual keyboard for Windows — built for users with severe motor disabilities who operate a touchscreen (wheelchair-mounted or school laptop) while applications run on the PC.
License: MIT Platform: Windows Build
Built with Tauri, React, Rust, and SQLite. An optional Android tablet companion provides trackpad, numpad, and remote typing over the local network.
Pre-built installers are on the Releases page.
- Download the latest
.msior.exeinstaller for Windows. - Run the installer. On older Windows builds, you may need the WebView2 runtime.
- Open the app on your accessibility touchscreen, select a profile in Settings, and position the window on the correct display.
The companion turns an Android tablet into a control surface (keyboard, trackpad, numpad, dictation, suggestions). The Windows host still injects keystrokes and mouse events into the focused app. Teaching / Music stay on Windows only.
Pairing stays in Settings → Companion (Start bridge / QR). The Companion mode tablet stays unavailable until the bridge is running or a tablet is paired. When a tablet connects, Companion becomes the selected mode and the host minimizes. Disconnect restores the previous mode and unminimizes but keeps the bridge running (reconnect without tapping Companion again). Tapping Normal, Mini, or Teaching — or Stop — stops the bridge.
Use a release installer or, for contributors, npm run tauri dev (the companion bridge can start from Settings).
Caregivers: download the pre-built APK from the public install page:
https://reachpanel-companion.vercel.app/
Scan this QR on the tablet to open that page, then tap Download APK:
(After the first Vercel deploy, update this URL and regenerate the QR if your project name differs.)
- Open the link (or scan the QR) on the tablet and tap Download APK.
- Allow installs from the browser if Android prompts (unknown apps).
- Install and open ReachPanel Companion.
Tablets are required. Phone-sized devices can install the APK but are blocked when the app launches.
Contributors: run from source with Expo Go or a local dev build — see companion/README.md (npx expo start, npx expo run:android, or EAS).
- Put the PC and tablet on the same local network.
- On Windows: ReachPanel → Settings → Companion.
- Click Start bridge (local WebSocket, typically port 17890).
- Show the QR if needed; use New pairing code to refresh it.
- On the tablet: scan the QR, or paste JSON via Copy pairing JSON on the host.
- Host session should show Active; the tablet shows keyboard / trackpad / numpad / dictation / suggestions.
- On the tablet: Settings → Network → Hotspot & tethering → USB tethering.
- Connect tablet ↔ PC with a data USB cable (not charge-only).
- On the host Companion page: New pairing code so the QR uses the USB network IP (often
192.168.42.xor192.168.137.x). - Scan the updated QR (or paste JSON) on the tablet — same protocol as Wi‑Fi.
- Already paired? A new QR updates the IP and reconnects when the host ID matches.
- Use the tablet for pointer (trackpad / numpad) and typing; the PC injects into the focused window.
- Tablet dictation uses the tablet mic. Groq API key (for languages without a Windows speech pack) stays in host Settings.
- Revoke tablets under Settings → Companion → paired devices.
- Bridge stopped → Start bridge again.
- Pairing fails → same LAN or USB tether; refresh with New pairing code; allow local inbound traffic if a firewall blocks the bridge port.
- Expo / audio module errors → companion/README.md troubleshooting.
| Main layout | Modes / Teaching | Settings / Companion |
|---|---|---|
| Main keyboard layout | Mini or Teaching | Settings + companion QR |
Replace with captures from your release build when ready — see docs/images/README.md.
- Normal — full typing keyboard and predictive suggestions
- Mini — compact keyboard; collapses to a branded FAB when idle; optional transparent outlined keys (white / dark gray / silver)
- Teaching — fullscreen work area; switch Language / Music / Math from the lesson section header. Music: piano/synth (2–5 octaves), partiture, built-in songs and file import, resizable lesson split. Language: Spelling (Play to spell from built-in or custom lists, optional Greek tone checking) and Free write (notepad + PDF library/viewer); Math is a placeholder for now
- System-wide keyboard injection with sticky modifiers
- Colors, opacity, and panel-driven key sizing
- Fn key on the bottom row maps number keys 1–0 and
-=to F1–F12 - Language switch key with country flag icons (follows installed Windows keyboards)
- On-screen layout override (auto / QWERTY / QWERTZ / AZERTY / Greek)
- Predictive text with offline word packs (English bundled; other languages downloadable), learns from typing, disable toggle
- Dictation key beside Right Ctrl (see Voice dictation below); show/hide the mic key under Settings → Visible sections
- Input preview — when an external text field is targeted, a live thumbnail of that field appears above the keyboard. Normal mode: Settings → Visible sections → Live input preview (Normal). Mini Mode: Settings → Mini Mode → Live input preview (Mini Mode). Each mode has its own toggle (both on by default). Disabled during Companion sessions. Protected or DRM content may appear black; some custom controls may not expose a usable screen region.
- Special keys use symbols (Enter ↵, Shift ⇧, Backspace ⌫, Space ␣)
- Right-click / long-press context menus are suppressed on the keyboard and main app surface
- Taskbar position — Settings moves the taskbar on the accessibility monitor (the display selected in Settings) between top and bottom only. On extended desktops each monitor can have its own taskbar edge; on mirrored/duplicate setups all copies of that display update together. Windows 10: top/bottom via StuckRects. Windows 11: top when the OS still honors StuckRects /
TaskbarLocation(verified via live tray geometry after Explorer restart). Builds that reject the move fail without a false success and may open Settings → Personalization → Taskbar. Explorer restarts when a change is attempted.
- Pair via Settings → Companion (QR, paste JSON, or USB tether)
- Tablet: keyboard, trackpad (tap-to-click), numpad, dictation, suggestions
- Host-only injection; Teaching / Music remain on Windows
- Dictate key on the keyboard (right of Right Ctrl)
- Uses the typing language, not the UI language
- English and Western European languages (DE, FR, IT, ES, PT): Windows Speech Recognition (WinRT) when the speech language pack is installed and online speech is allowed
- Greek and other languages without a Windows speech pack: Groq cloud Whisper (
whisper-large-v3-turbo) over the internet - After the first Groq use that local day, the key can show remaining daily request % (RPD)
- Create a free account at console.groq.com.
- Open API Keys and create a key (starts with
gsk_…). - In ReachPanel, open Settings.
- Under the keyboard / dictation options, paste the key into Groq API key (cloud dictation).
- Set typing language to Greek (EL), then use the dictate key on the keyboard.
You can also set the GROQ_API_KEY environment variable instead of (or as a fallback for) the Settings field. An internet connection is required for cloud dictation. WinRT dictation for English and Western European languages does not need a Groq key.
If Windows online speech is off, open Settings → Privacy & security → Speech and turn on Online speech recognition. For speech packs, use Settings → Time & language → Speech.
- Named file-based profiles (create, save, load, wipe)
- Multi-monitor detection and positioning
- Color profiles and optional background image
- In-app update check and prompt
- About section (version and links)
- UI languages: English, Greek, German, French, Italian, Spanish, Portuguese
Windows 10 or 11 only. ReachPanel is not supported on macOS — touchscreens on macOS expose only single-point touch, which breaks the multi-touch layout and monitor targeting this app requires.
Companion: Android tablet on the same LAN as the PC, or USB tethering. See Install & connect the Android companion.
For development:
- Node.js 18+ (Node 20+ for the companion Expo app)
- Rust (via rustup)
- Visual Studio Build Tools with the C++ workload
npm install
npm run tauri devFor Rust builds on Windows, ensure the MSVC environment is active. If link.exe is not found, run from a Developer Command Prompt or use:
"C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Auxiliary\Build\vcvars64.bat"Companion app: companion/README.md.
The caregiver APK page lives at docs/install/index.html and deploys to Vercel. CI replaces __INSTALL_APK_PUBLIC_URL__ with the Blob URL before publish.
One-time GitHub Actions secrets (Settings → Secrets and variables → Actions):
| Secret | Purpose |
|---|---|
EXPO_TOKEN |
EAS build auth |
EAS_PROJECT_ID |
Expo project (run eas init inside companion/ — do not invent a UUID) |
VERCEL_TOKEN |
Vercel deploy |
VERCEL_ORG_ID |
Vercel team / user |
VERCEL_PROJECT_ID |
Vercel project for docs/install |
BLOB_READ_WRITE_TOKEN |
Upload APK to Vercel Blob |
Optional repository variable: INSTALL_APK_PUBLIC_URL (override the default Blob URL if needed).
Before the first CI run, inside companion/: eas init and eas credentials for Android signing.
npm run tauri buildInstallers and executables are written to:
src-tauri/target/release/bundle/
Windows installers are built in CI and published automatically when a release-please Release PR is merged on main. Development happens on dev; merging dev → main opens the Release PR. See CONTRIBUTING.md for the full branch and versioning workflow.
| Path | Responsibility |
|---|---|
src/ |
React frontend (keyboard, modes, teaching, settings, companion UI) |
src-tauri/src/input/ |
Keyboard/mouse injection (Windows SendInput) |
src-tauri/src/window/ |
Monitor enumeration and taskbar-aware layout (Windows APIs) |
src-tauri/src/tts/ |
Text-to-speech (Windows SAPI / WinRT) |
src-tauri/src/stt/ |
Voice dictation (WinRT speech recognition; Groq cloud fallback) |
src-tauri/src/db/ |
SQLite persistence (profiles, phrases, macros) |
src-tauri/src/prediction/ |
Predictive text + word-pack install/download |
src-tauri/src/companion/ |
Android tablet WebSocket bridge (pairing, session, dispatch) |
src-tauri/src/profiles/ |
File-based named profiles |
companion/ |
Expo Android companion app |
src-tauri/resources/wordpacks/ |
Bundled English dictionary |
wordpacks-dist/ |
Downloadable packs for GitHub wordpacks-v1 release |
docs/accessibility-requirements.md |
Target users and acceptance criteria |
- Eye tracking support
- AutoHotkey macro import
- iOS companion (TestFlight)
- Remove leftover Windows chrome for mouse panel, phrases, Quick Actions, macros, and head tracking (tablet trackpad/numpad stays)
- Input injection does not work into elevated (admin) applications or UAC prompts
- Voice dictation is Windows-only on the host; Greek (and other WinRT-unsupported languages) need a Groq API key and internet
- Cloud dictation is utterance-based (speak, pause) rather than fully continuous streaming
- Companion needs the same LAN or USB tether; Teaching / Music are not available on the tablet
Contributions are welcome. See CONTRIBUTING.md and docs/accessibility-requirements.md before opening a pull request.
Please read our Code of Conduct.
To report a security vulnerability, see SECURITY.md.
MIT — see LICENSE.
This is assistive software, not a medical device. Test thoroughly with your own hardware and accessibility setup before relying on it in daily use.