Skip to content

UI evaluators: app driver over the automation socket and host-launched test target - #262

Open
MHohlios wants to merge 7 commits into
mainfrom
wiffletree/ui-evaluators-app-driver-over-the-automation-soc
Open

UI evaluators: app driver over the automation socket and host-launched test target#262
MHohlios wants to merge 7 commits into
mainfrom
wiffletree/ui-evaluators-app-driver-over-the-automation-soc

Conversation

@MHohlios

@MHohlios MHohlios commented Nov 4, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

This is the second of the stacked PRs for the OpenSpec change ui-evaluators, on top of #261, which is now on main. App-only evaluator runs can now drive the app under test, and only that app.

  • Test copy started by the host. start_evaluation in app mode takes a target: an absolute executable, arguments with one --data-dir <existing folder>, and an environment. Before each turn the host launches the copy with --automation-socket on a short socket path for the run.
    • The copy's environment is cleared, the given PATH goes first, and WORKSPACE_SKIP_LOGIN_ENV=1 makes its host skip login-shell PATH adoption. No WORKSPACE_* or *_TOKEN variable reaches it.
    • The host waits for the socket and fails the run with the reason if it never answers.
    • When the turn ends, the host stops the copy and everything it started. A process watcher keeps helpers that were reparented after the copy exited. The copy is reaped only after its group has been signalled. A followed group is signalled only while a recorded member is still alive in it. Together these rule out a reused process id.
  • Data-folder safety. The data folder must already exist and is compared by its real path, so links and .. can't get around the checks. A data folder that overlaps Wiffletree's own data is refused. So are a second unended run on the same folder, a repeated --data-dir, and the --data-dir=<path> form.
  • App driver. wiffletree-host app-driver-mcp is the run's only MCP server, named app. Its tools are screenshot, click, double_click, right_click, move, type_text, key, scroll, drag, resize_window and window. Coordinates are window points. Screenshots are saved only inside the run folder; links out of it are refused, and the file is opened with O_NOFOLLOW.
  • Screenshots without Screen Recording. A new in-app capture step gets the app's own window from CGWindowListCreateImage, at 1× and at the display's native scale. This was measured from a launch with responsibility disclaimed (CGPreflightScreenCaptureAccess = 0): the capture worked while screencapture -l failed. scripts/drive_desktop.py shot now uses this step.
  • Launch checks. Claude and Codex launches gain only the app server and exactly its tools. The start and tool-call checks reject anything else.
  • CI. The mac nextest profile now selects the new macOS-only provider::app_target tests.

Decisions

  • The coordinator creates the test copy's data folder before the run starts. This replaces resolving folders that don't exist yet, which two review rounds showed could be bypassed.
  • Claude shrinks images larger than 2000 px on the long edge before the model sees them. The tool description says to sample pixels from a 1× shot or a smaller native shot.

How it was tested

  • cargo fmt --check and cargo clippy --workspace --all-targets -D warnings are clean, and cargo test --workspace passes.
  • New tests cover:
    • the copy's arguments and environment;
    • a socket that never answers, a copy that exits, and a stop during startup;
    • cleanup, including a helper in its own session after the copy exited, and a copy that stays unreaped until its group has been signalled;
    • the target refusals and the same-data-folder refusal, including links and ..;
    • the driver schemas and the socket steps each tool maps to;
    • screenshot containment;
    • the Claude and Codex launch and start checks;
    • a full service turn with a fake provider and a stand-in app.
  • The app_target and service tests ran 20 times in a loop, and 5 more times under the mac nextest profile. None hung, and no stray processes were left.
  • By hand:
    • the real driver drove a real built workspace-desktop through every tool, including both screenshot scales and resizes to 960×600, the largest size and 1280×820;
    • a reviewer ran the real Claude Code 2.1.317 with the evaluator's flags and the real driver. It saw exactly the 5 file tools and the 11 mcp__app__* tools, and the image reached the model.

Follow-ups

  • Window geometry and in-app capture have no unit tests (crates/workspace-desktop/src/automation.rs:249).
  • Re-confirm on this Mac that in-app capture works once wiffletree-host no longer holds Screen Recording (openspec/changes/ui-evaluators/specs/ui-evaluators/spec.md:92).
  • The driver's MCP stdio loop has no end-to-end test (crates/workspace-host/src/app_driver.rs:262).
  • No test covers what the login-env skip switch does in main.rs; tests only check that the launcher sets it (crates/workspace-host/src/main.rs:173).
  • The test copy isn't recorded with the run's processes, so a host crash can leave it running (crates/workspace-host/src/provider/app_target.rs:48).

Reviewed with Wiffletree: 2 full reviews and 2 re-checks, 6 rounds (Tests, Claude, Codex, Style)

🤖 Generated with Claude Code

Base automatically changed from wiffletree/ui-evaluators-isolated-model-pinned-evaluator-ru to main November 4, 2026 15:26
mhohlios added 6 commits November 4, 2026 10:27
mhohlios
Drive app-only evaluator runs through a host-launched test copy
99d3e75 
An app run's start_evaluation now names its target: the app's executable,
arguments with a test data folder of its own, and the variables it needs.
For each turn, the host starts that copy with --automation-socket on a
socket of the run's own. Its environment is cleared and holds no session
variable or token, a given PATH goes first, and WORKSPACE_SKIP_LOGIN_ENV
keeps the copy's host from adopting the login shell's PATH. The provider
starts only once the socket answers; a copy that exits or never answers
fails the run with the reason. However the turn ends, the copy's group
and every group it started are stopped. Two unended runs cannot share a
data folder, and none may use Wiffletree's own.

The run's only MCP server is the app driver, wiffletree-host
app-driver-mcp: screenshot, click, double_click, right_click, move,
type_text, key, scroll, drag, resize_window and window, in window points.
The Claude and Codex launches and their start and tool-call checks allow
exactly that server and those tools.

The automation socket gains capture, which renders the app's own window
through CGWindowListCreateImage at points or native scale, and resize,
which keeps the window within its minimum and its display. Measured on
macOS 27.0.1 from a launch with no Screen Recording grant: the app
captured its own window, while screencapture -l failed. drive_desktop.py's
shot uses it, so it no longer needs the permission.
mhohlios
Save driver screenshots without following links out of the run folder
ebf1540 
The driver runs outside the evaluator's sandbox, so a link already in the
run folder could lead a screenshot's write, or the folders created for it,
out of the folder. Each existing part of the path must not be a link, and
the file opens with O_NOFOLLOW.
mhohlios
Resolve a test copy's data folder as the app will, and allow one --da…
eccbee8 
…ta-dir

workspace-desktop uses the last --data-dir it is given, so a target with
several could open a folder the checks never saw; only one is allowed now.
A profile not created yet is resolved from its deepest existing ancestor
with . and .. applied, so two spellings of one new folder are the same
folder to the same-data-folder refusal.
mhohlios
Require a test copy's data folder to exist and compare it canonically
3e3b1a1 
The coordinator now creates the data folder before start_evaluation, and
Wiffletree canonicalizes it, so links and .. cannot hide an overlap with
its own folder or a folder another run uses; the partial resolution of
folders not yet created is gone. The --data-dir=<path> spelling is
refused, so the only data folder checked is the one the app reads.

The screenshot tool says Claude downsizes images over 2000 pixels on the
long edge, so pixels are sampled from a points shot. The design states
the driver as Codex's one MCP exception.
mhohlios
Stop what a test copy started after the copy itself exited
ebf76ad 
The stop listed only the copy's current descendants, so a helper in a
session of its own lost its parent when the copy exited, at startup or
later, and was never signalled. The copy's tree is now followed by the
process watcher, which keeps orphaned members, and the stop signals every
group it followed.
mhohlios
Run the test copy's macOS-only tests in the mac nextest profile
514340f
@MHohlios
MHohlios force-pushed the wiffletree/ui-evaluators-app-driver-over-the-automation-soc branch from 2db6620 to 514340f Compare November 4, 2026 15:37
mhohlios
Keep a test copy unreaped until its stop, and confirm each group befo…
7bc90cb 
…re signalling it

The copy's exit check and the stop's grace loop reaped it, after which its
id, and its group's, could name an unrelated process the stop still
signalled. The exit check now leaves it unreaped, the copy's group is
stopped as a provider's is, frozen while the watcher looks, and each group
of what it started is stopped with stop_group, only while a process the
watcher followed is alive in it. A test waiting for a pid file is bounded.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@MHohlios