-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathgershwin-automated-testing.html
More file actions
339 lines (318 loc) · 42.7 KB
/
Copy pathgershwin-automated-testing.html
File metadata and controls
339 lines (318 loc) · 42.7 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta http-equiv="Cache-Control" content="no-cache, no-store, must-revalidate">
<meta http-equiv="Pragma" content="no-cache">
<meta http-equiv="Expires" content="0">
<title>Native automated testing for GNUstep / Gershwin — every category, options & gaps</title>
<style>
:root{
--fg:#1f2328; --fg-soft:#57606a; --bg:#ffffff; --bg-soft:#f6f8fa;
--border:#d0d7de; --accent:#0969da; --accent-dark:#0a4ea3;
--green:#1a7f37; --red:#cf222e; --amber:#9a6700; --code-bg:#f6f8fa;
}
*{box-sizing:border-box;}
html,body{margin:0;padding:0;}
body{font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Helvetica,Arial,sans-serif;
color:var(--fg);background:var(--bg);line-height:1.6;-webkit-font-smoothing:antialiased;}
.wrap{max-width:940px;margin:0 auto;padding:40px 24px 100px;}
header.masthead{border:1px solid var(--border);border-radius:8px;padding:28px 28px;margin-bottom:36px;background:var(--bg-soft);}
.logo{font-size:12px;letter-spacing:3px;color:var(--accent);text-transform:uppercase;margin-bottom:12px;font-weight:600;}
h1{font-size:28px;line-height:1.2;margin:0 0 12px;color:var(--fg);letter-spacing:-0.3px;}
.subtitle{color:var(--fg-soft);font-size:15px;max-width:80ch;}
.meta-row{margin-top:20px;display:flex;flex-wrap:wrap;gap:8px;font-size:12px;}
.tag{border:1px solid var(--border);border-radius:999px;padding:3px 11px;color:var(--fg-soft);letter-spacing:0.5px;background:#fff;}
h2{font-size:21px;color:var(--fg);margin:52px 0 10px;letter-spacing:-0.2px;padding-bottom:6px;border-bottom:1px solid var(--border);}
h2 .num{display:inline-block;color:#fff;background:var(--accent);border-radius:6px;padding:1px 10px;margin-right:12px;font-size:16px;font-weight:600;}
h3{font-size:16px;color:var(--fg);margin:30px 0 6px;}
.lead{color:var(--fg);max-width:84ch;margin:0 0 8px;font-size:15px;}
.lead.dim{color:var(--fg-soft);}
p,li{font-size:15px;}
a{color:var(--accent);text-decoration:none;}
a:hover{text-decoration:underline;}
b,strong{color:var(--fg);font-weight:600;}
em{color:var(--fg);font-style:italic;}
table.spec{width:100%;border-collapse:collapse;margin:16px 0;font-size:13px;}
table.spec th,table.spec td{border:1px solid var(--border);padding:6px 9px;text-align:left;vertical-align:top;}
table.spec th{color:var(--fg);background:var(--bg-soft);font-weight:600;}
table.spec td{color:var(--fg);}
table.spec tr:nth-child(even) td{background:#fbfcfd;}
table.spec code{color:var(--accent-dark);}
table.matrix td{text-align:center;}
table.matrix td:first-child{text-align:left;}
.scroll{overflow-x:auto;}
.y{color:var(--green);font-weight:700;}
.n{color:var(--red);font-weight:700;}
.p{color:var(--amber);font-weight:700;}
ul.notes{margin:12px 0 0;padding:0;list-style:none;display:grid;gap:8px;}
ul.notes li{border:1px solid var(--border);border-left:4px solid var(--accent);border-radius:4px;padding:9px 14px;font-size:14px;color:var(--fg);background:var(--bg-soft);}
ul.notes li b{color:var(--fg);}
ul.notes li.warn{border-left-color:var(--red);background:#fff5f5;}
ul.notes li.warn b{color:var(--red);}
ul.notes li.ok{border-left-color:var(--green);background:#eafbef;}
ul.notes li.ok b{color:var(--green);}
.code{margin:18px 0 8px;background:var(--code-bg);border:1px solid var(--border);border-radius:6px;position:relative;overflow-x:auto;}
.code .code-label{position:absolute;top:0;right:0;background:#eaeef2;color:var(--fg-soft);font-size:10px;letter-spacing:1px;padding:3px 9px;border-bottom-left-radius:6px;border-left:1px solid var(--border);border-bottom:1px solid var(--border);}
pre.code-body{margin:0;padding:26px 18px 18px;font-family:ui-monospace,"SF Mono","JetBrains Mono",Menlo,Consolas,monospace;font-size:12.5px;line-height:1.5;color:var(--fg);white-space:pre;}
pre.code-body .c{color:#6e7781;}
pre.code-body .k{color:#0550ae;}
pre.code-body .s{color:var(--green);}
code.inl{font-family:ui-monospace,"SF Mono",Menlo,Consolas,monospace;font-size:0.88em;color:var(--accent-dark);background:#eff1f3;padding:1px 5px;border-radius:4px;}
.flow{display:flex;flex-wrap:wrap;align-items:center;gap:6px;margin:18px 0 4px;font-size:12.5px;}
.flow .node{border:1px solid var(--border);border-radius:6px;padding:7px 12px;background:var(--bg-soft);white-space:nowrap;}
.flow .arr{color:var(--fg-soft);}
.flow .node.gate{border-color:var(--red);color:var(--red);background:#fff5f5;}
.flow .node.ok{border-color:var(--green);color:var(--green);background:#eafbef;}
.pyr{max-width:560px;margin:14px auto;font-size:12.5px;}
.pyr .row{display:flex;align-items:center;gap:10px;margin:4px 0;}
.pyr .bar{border:1px solid var(--border);border-radius:5px;padding:6px 10px;background:var(--bg-soft);text-align:center;}
footer{margin-top:70px;padding-top:22px;border-top:1px solid var(--border);color:var(--fg-soft);font-size:13px;}
.srcs{font-size:12.5px;color:var(--fg-soft);columns:2;column-gap:28px;}
.srcs li{margin-bottom:6px;break-inside:avoid;}
@media (max-width:640px){.srcs{columns:1;}}
</style>
</head>
<body>
<div class="wrap">
<header class="masthead">
<div class="logo">Gershwin · QA / Testing</div>
<h1>Native automated testing for GNUstep / Gershwin: every category, the options, and the gaps</h1>
<p class="subtitle">What native mechanism covers each layer of testing — <b>unit, integration, functional, UI, acceptance/BDD, system/smoke, and regression</b> — for a GNUstep/AppKit stack. Specifically: what <b>UnitKit</b> and the GNUstep <b>Testing</b> framework cover (unit/integration), what <b>StepTalk</b> and <b>Distributed Objects</b> cover (functional/integration driving), where UI/acceptance testing has <b>no native answer</b> (GNUstep has no working accessibility layer), and how the current <code class="inl">gershwin-on-nextbsd</code> screenshot gate fits (system/smoke). Ends with a proposed native <b>Gherkin/BDD</b> framework and a full-pyramid strategy that runs in GitHub Actions and locally.</p>
<div class="meta-row">
<span class="tag">TESTING LANDSCAPE</span>
<span class="tag">UNIT · INTEGRATION · FUNCTIONAL</span>
<span class="tag">UI · BDD · SYSTEM</span>
<span class="tag">GNUSTEP / APPKIT</span>
<span class="tag">CI + LOCAL</span>
<span class="tag">DISTRIBUTED OBJECTS · NO D-BUS</span>
</div>
</header>
<p class="lead">Output of a multi-agent investigation into GNUstep-native testing plus a live read of the Gershwin ISO test scripts. Every capability claim is grounded in a cited source (§11) or actual repo code. <b>Terminology note:</b> there is <em>no "UIKit" in GNUstep</em> — UIKit is Apple's iOS framework; GNUstep's GUI framework is <b>AppKit</b> (<code class="inl">libs-gui</code>). The similarly-named <b>UnitKit</b> is Étoilé's <em>unit-test</em> framework (§2) — a different thing entirely.</p>
<!-- =============== 1 =============== -->
<h2><span class="num">1</span>The test-category map — what covers what</h2>
<p class="lead">First, a plain taxonomy (your categories, lightly standardized — they were close):</p>
<div class="scroll">
<table class="spec">
<tr><th>Category</th><th>Question it answers</th><th>Scope</th></tr>
<tr><td><b>Unit</b></td><td>Does this one class/method behave?</td><td>1 object, in-process, no I/O</td></tr>
<tr><td><b>Integration</b></td><td>Do these components work together?</td><td>≥2 components / processes (e.g. a framework + its deps, or DO across processes)</td></tr>
<tr><td><b>Functional</b></td><td>Does a feature work end-to-end at the behavior/API level?</td><td>A whole feature, often headless (no GUI needed)</td></tr>
<tr><td><b>UI</b></td><td>Does the actual GUI respond to clicks/keys correctly?</td><td>Real widgets on a real (or virtual) display</td></tr>
<tr><td><b>Acceptance / BDD</b></td><td>Does it do what a human spec says, in plain language?</td><td>Feature described in Given/When/Then; usually rides on functional or UI</td></tr>
<tr><td><b>System / smoke</b></td><td>Does the whole thing boot and come up?</td><td>The full ISO in a VM</td></tr>
<tr><td><b>Regression</b></td><td>Did a past bug come back?</td><td>Cross-cutting — any of the above, re-run on every change; visual regression = golden-image diff</td></tr>
</table>
</div>
<p class="lead">The coverage matrix — the core answer. <span class="y">✔</span> = strong native fit · <span class="p">~</span> = possible/partial · <span class="n">✗</span> = not this mechanism.</p>
<div class="scroll">
<table class="spec matrix">
<tr><th style="text-align:left">Mechanism (native unless noted)</th><th>Unit</th><th>Integ.</th><th>Funct.</th><th>UI</th><th>Accept.</th><th>System</th><th>Regr.</th></tr>
<tr><td><b>GNUstep Testing framework</b> (<code>make check</code>)</td><td class="y">✔</td><td class="y">✔</td><td class="p">~</td><td class="n">✗</td><td class="n">✗</td><td class="n">✗</td><td class="y">✔</td></tr>
<tr><td><b>Étoilé UnitKit</b></td><td class="y">✔</td><td class="p">~</td><td class="p">~</td><td class="n">✗</td><td class="n">✗</td><td class="n">✗</td><td class="y">✔</td></tr>
<tr><td><b>swift-corelibs-xctest</b></td><td class="y">✔</td><td class="p">~</td><td class="p">~</td><td class="n">✗</td><td class="n">✗</td><td class="n">✗</td><td class="y">✔</td></tr>
<tr><td>ObjC BDD-unit (Kiwi/Cedar/Specta)¹</td><td class="y">✔</td><td class="p">~</td><td class="n">✗</td><td class="n">✗</td><td class="p">~</td><td class="n">✗</td><td class="p">~</td></tr>
<tr><td><b>Distributed Objects</b> (harness transport)</td><td class="n">✗</td><td class="y">✔</td><td class="y">✔</td><td class="p">~²</td><td class="p">~</td><td class="n">✗</td><td class="p">~</td></tr>
<tr><td><b>StepTalk</b> (scripting driver)</td><td class="n">✗</td><td class="y">✔</td><td class="y">✔</td><td class="p">~²</td><td class="p">~</td><td class="n">✗</td><td class="p">~</td></tr>
<tr><td><b>NSEvent synthesis + in-app agent</b></td><td class="n">✗</td><td class="p">~</td><td class="y">✔</td><td class="y">✔</td><td class="y">✔</td><td class="n">✗</td><td class="p">~</td></tr>
<tr><td>External X11 (xdotool/SikuliX/PyAutoGUI)</td><td class="n">✗</td><td class="n">✗</td><td class="p">~</td><td class="y">✔</td><td class="p">~</td><td class="p">~</td><td class="p">~</td></tr>
<tr><td><b>Gherkin/BDD framework</b> (proposed, §6)</td><td class="n">✗</td><td class="p">~</td><td class="y">✔</td><td class="y">✔</td><td class="y">✔</td><td class="p">~</td><td class="y">✔</td></tr>
<tr><td>QEMU screenshot gate (have it, §7)</td><td class="n">✗</td><td class="n">✗</td><td class="p">~</td><td class="n">✗³</td><td class="p">~</td><td class="y">✔</td><td class="p">~⁴</td></tr>
<tr><td>ImageMagick <code>compare</code> (golden image)</td><td class="n">✗</td><td class="n">✗</td><td class="n">✗</td><td class="p">assert</td><td class="n">✗</td><td class="p">~</td><td class="y">✔⁴</td></tr>
</table>
</div>
<p class="lead dim">¹ Apple/XCTest-oriented; <b>GNUstep support unverified</b> — treat as "maybe." ² Only by having the bridged object post synthesized NSEvents (§5). ³ Whole-screen heuristic, blind to individual widgets. ⁴ The current gate uses a colour-count heuristic, <em>not</em> a golden-image diff — true visual regression would add <code class="inl">compare</code> (§9).</p>
<ul class="notes">
<li class="ok"><b>The one-sentence shape of the landscape:</b> GNUstep is <b>strong at unit/integration</b> (a real xUnit stack: the Testing framework, UnitKit, XCTest) and has a <b>native functional/integration <em>driver</em></b> (StepTalk / Distributed Objects that reach into a running app to invoke actions and read state) — but it has <b>no native UI, acceptance, or visual-regression tooling</b>, because it has no working accessibility layer (§4). Those top layers are what you build (§5–§6) or approximate with pixels/OCR.</li>
<li><b>Direct answers to your question:</b> <b>UnitKit</b> covers <em>unit</em> tests (xUnit assertions; functional/integration only as far as the code you put in a test body reaches) — it never touches the GUI. <b>StepTalk</b> covers <em>functional/integration</em> by <em>scripting a live app</em> (invoke actions, read object state) — it is a driver, not a unit framework, and does no unit testing.</li>
</ul>
<!-- =============== 2 =============== -->
<h2><span class="num">2</span>Unit tests (native — the strong layer)</h2>
<div class="scroll">
<table class="spec">
<tr><th>Framework</th><th>Style</th><th>What it covers</th><th>Maturity / notes</th></tr>
<tr><td><b>GNUstep "Testing" framework</b> (gnustep-tests)</td><td>macro-based (<code>PASS</code>, <code>PASS_EQUAL</code>, <code>PASS_EXCEPTION</code>)</td><td>Unit + integration of ObjC classes; <b>this is the harness GNUstep itself uses to regression-test libs-base/libs-gui</b></td><td>Ships in <b>gnustep-make</b> (<code>$GNUSTEP_MAKEFILES/TestFramework/</code>, <code>Testing.h</code>, <code>ObjectTesting.h</code>, <code>gnustep-tests</code> runner); wired via the <code>make check</code> target. The default, canonical choice.</td></tr>
<tr><td><b>Étoilé UnitKit</b></td><td>xUnit (<code>UKRunner</code>, <code>UKTest</code> marker, <code>UK*</code> asserts)</td><td>Unit; integration/functional as far as the test body drives</td><td>Minimalist (<2000 LoC), Mac/iOS/GNUstep; Étoilé ships a GNUstep port. <b>This is what "UnitKit" refers to</b> — a unit framework, not "UIKit".</td></tr>
<tr><td><b>swift-corelibs-xctest</b></td><td>XCTest (<code>XCTAssert*</code>)</td><td>Unit; functional if written so</td><td>Swift-on-Foundation; runs on GNUstep Foundation but off Apple's ObjC runtime needs explicit <code>allTests</code>/<code>XCTMain</code>. No XCUITest (the GUI half is Apple-private).</td></tr>
<tr><td>Kiwi / Cedar / Specta+Expecta / Quick</td><td>RSpec-style BDD-<em>unit</em> (<code>describe</code>/<code>it</code>)</td><td>Behavior specs at the unit level</td><td><b>Apple/XCTest-bound; GNUstep compatibility unverified</b> — likely needs porting. Note this is BDD at the <em>unit</em> level (describe/it), distinct from Gherkin acceptance specs (§6).</td></tr>
</table>
</div>
<ul class="notes">
<li class="ok"><b>Recommendation for Gershwin components:</b> use the <b>GNUstep Testing framework</b> (<code class="inl">make check</code>) — it's the native default, already the regression harness for the core libraries, and needs no extra dependency. Reach for UnitKit only if you want its xUnit ergonomics across an Étoilé-style tree.</li>
<li><b>These are in-process:</b> they instantiate objects and assert on them. They never drive a live GUI — that's a different category (§5).</li>
</ul>
<!-- =============== 3 =============== -->
<h2><span class="num">3</span>Integration & functional tests (native driving)</h2>
<p class="lead">Above the unit line, GNUstep's advantage is that <b>Distributed Objects</b> lets a test process reach into a <em>running</em> app and message its real objects — the native substrate for integration and functional tests. StepTalk is a scripting front-end on top of it.</p>
<ul class="notes">
<li class="ok"><b>Distributed Objects (NSConnection/NSDistantObject) — the transport.</b> An app vends its <code class="inl">NSApplication</code> delegate under the app name by convention (the same mechanism NSServices uses), so a harness can <code class="inl">rootProxyForConnectionWithRegisteredName:host:</code> in and invoke methods / read state — full semantic control across a process boundary. Category: <b>integration + functional</b>. Gap: you drive whatever the delegate exposes; there's no standard "test object," and no known off-the-shelf DO test harness — you'd build a thin one.</li>
<li class="ok"><b>StepTalk — the scripting driver.</b> A language-independent (default Smalltalk) scripting framework that exposes a live app's object graph to scripts: invoke actions, set/read values, walk the model. Category: <b>functional / integration / lightweight acceptance</b>. It is <em>not</em> a unit framework and does no assertions of its own — you assert inside the script. An app becomes scriptable by shipping a <code class="inl">ScriptingInfo.plist</code>; StepTalk can't script an un-instrumented app. Implemented and actively used (gs-desktop treats it as first-class), but niche (latest 0.10.0).</li>
<li><b>NSServices — narrow IPC.</b> Invoke a <em>declared</em> service via <code class="inl">NSPerformService()</code> and read its pasteboard result. A functional/integration data path for the specific operations an app exposes as services — not a general driver.</li>
<li class="warn"><b>What "functional, no GUI" looks like in practice:</b> spin the component up in a <code class="inl">make check</code> test (Testing framework) or drive a running app over DO/StepTalk, then assert on returned state. This is the most robust automated testing available on GNUstep today because it never touches pixels.</li>
</ul>
<!-- =============== 4 =============== -->
<h2><span class="num">4</span>The UI / acceptance wall: no accessibility</h2>
<p class="lead">Everything above is headless and native. The moment you want to test the <em>GUI</em>, you hit the wall that shapes the rest of this document.</p>
<ul class="notes">
<li class="warn"><b>GNUstep has no working accessibility layer.</b> libs-gui's <code class="inl">NSAccessibility.m</code> is ~95% the constant <em>vocabulary</em> (hundreds of role/attribute/notification/action name strings — real and reusable as keys) plus a handful of free functions that are <b>all non-functional</b>: <code class="inl">NSAccessibilityPostNotificationWithUserInfo</code> is an empty <code class="inl">// FIXME</code> (so no accessibility notifications ever fire) and the rest (<code class="inl">UnignoredAncestor</code>/<code class="inl">Descendant</code>/<code class="inl">Children</code>, <code class="inl">RoleDescription</code>, <code class="inl">ActionDescription</code>) just <code class="inl">return nil</code>. Worse, the <em>per-widget</em> <code class="inl">NSAccessibility</code> methods on the AppKit classes are only thinly present (the element classes were added ~0.29.0, largely as skeletons) — so nothing publishes a usable tree or emits change events. There is <b>no AT-SPI2/D-Bus bridge, and Gershwin doesn't want one</b> (§4.1). The AT-SPI ecosystem covers GTK, Java/Swing, Mozilla, OpenOffice; GNUstep is absent.</li>
<li class="warn"><b>Consequence:</b> accessibility-driven tools (<b>LDTP, Dogtail, Accerciser</b>) are <em>blocked</em> — nothing to introspect. And "read the screen after input" becomes a <b>pixel/OCR problem, not a semantic one</b>: you locate controls by coordinates, image template-match, or OCR, and assert the same way. This is upstream in GNUstep, not fixable in the Gershwin ISO tooling — but the in-app-agent approach (§5–§6) routes <em>around</em> it, and §4.1 is the proper native fix.</li>
</ul>
<h3>4.1 · The Gershwin-native fix: NSAccessibility over Distributed Objects (no D-Bus)</h3>
<p class="lead"><b>AT-SPI2 and D-Bus bridges are explicitly out of scope for Gershwin.</b> AT-SPI2 is the Linux accessibility layer, and its "2" literally means "rewritten to run over <b>D-Bus</b>" — it stands up a separate a11y bus plus a registry daemon (<code class="inl">at-spi2-registryd</code>) and marshals every widget into a language-neutral D-Bus <em>Accessible</em> struct. Unwanted here. The correct fix is the one <b>macOS itself uses</b>: <code class="inl">NSAccessibility</code> is a Cocoa <em>object</em> API and on macOS does <em>not</em> use D-Bus at all (it rides Apple's own IPC). Gershwin implements that same API and transports it over <b>Distributed Objects</b> — Cocoa-authentic, D-Bus-free, and the same object layer that powers native UI/BDD testing.</p>
<ul class="notes">
<li class="ok"><b>DO is the more natural transport, not a compromise.</b> The accessibility tree is <em>already</em> Objective-C objects — <code class="inl">NSView</code>/<code class="inl">NSCell</code>/<code class="inl">NSControl</code> already carry role, title, value, enabled, and focus. DO vends them (or a thin facade) and a client sends real ObjC messages; you skip the entire marshalling-into-a-neutral-struct step that AT-SPI2 requires.</li>
</ul>
<div class="scroll">
<table class="spec">
<tr><th>AT-SPI2 / D-Bus concept</th><th>Gershwin DO-native equivalent</th></tr>
<tr><td>a11y <b>bus daemon</b> (<code>at-spi2-registryd</code>)</td><td><b><code>gdomap</code></b> — GNUstep's existing DO name/port registry. No new bus.</td></tr>
<tr><td>well-known bus <b>names</b></td><td><code>NSConnection registerName:</code> per app (extends the delegate-vending convention apps already use)</td></tr>
<tr><td><b>Accessible</b> object (role/state/name, marshalled)</td><td>the real <code>NSView</code>/<code>NSCell</code>/<code>NSControl</code>, vended — role/enabled/value/title already live on it</td></tr>
<tr><td>D-Bus <b>method call</b> (<code>Action.doAction</code>)</td><td>an ordinary ObjC message over DO: <code>[element performAction: …]</code></td></tr>
<tr><td>D-Bus <b>signals</b> (focus-/value-changed)</td><td><code>NSDistributedNotificationCenter</code>, or a client callback proxy the app messages on events (DO is bidirectional); implement <code>NSAccessibilityPostNotification</code> to route here</td></tr>
<tr><td>D-Bus <b>XML introspection</b></td><td>the <code>@protocol</code> itself — compile-time typed, no introspection needed</td></tr>
<tr><td>bus <b>service activation</b></td><td><code>launchd</code> (already in the Gershwin/NextBSD stack)</td></tr>
</table>
</div>
<ul class="notes">
<li><b>One layer, two consumers:</b> a screen reader <em>and</em> a Dogtail-equivalent test/BDD harness consume the same DO-vended tree. The in-app TestAgent (§6) is a client of this — so building it delivers <b>real accessibility as a side effect</b>, the Gershwin way.</li>
<li class="warn"><b>Does not exist yet — a modest greenfield worth building</b>, and less than it sounds because the hardest naming part is <em>already done</em>: <code class="inl">NSAccessibility.m</code> already declares the full constant vocabulary (roles/attributes/actions/notifications), so there's no namespace to design — only behavior to add. Two concrete tasks: <b>(1)</b> implement the per-widget attribute/action methods on the AppKit classes — the same view-tree walk the TestAgent already does — and <b>(2)</b> implement <code class="inl">NSAccessibilityPostNotification</code> (currently an empty <code class="inl">// FIXME</code> no-op) to fan out over DO / <code class="inl">NSDistributedNotificationCenter</code>. Then vend the tree over DO and register via <code class="inl">gdomap</code>. <b>No external D-Bus dependency</b> — the whole loop is Gershwin apps ↔ Gershwin tools, so there's nothing to shed.</li>
</ul>
<!-- =============== 5 =============== -->
<h2><span class="num">5</span>UI tests — native and external</h2>
<p class="lead">Two ways to drive real widgets: from <em>inside</em> the app (native NSEvents, semantic) or from <em>outside</em> at the X11/VM layer (toolkit-agnostic, pixel-based).</p>
<h3>5.1 · Native, in-process: NSEvent synthesis</h3>
<ul class="notes">
<li><b>Works:</b> build events with <code class="inl">+[NSEvent mouseEventWithType:location:…]</code> / <code class="inl">+keyEventWithType:…</code> and post via <code class="inl">-[NSApplication postEvent:atStart:]</code> → <code class="inl">sendEvent:</code>; they flow through normal hit-testing. Read the result by inspecting <code class="inl">NSControl</code>/<code class="inl">NSView</code>/model state — semantic, not pixels.</li>
<li class="warn"><b>Limitation:</b> app-level events only — an external process can't post them without being <em>inside</em> the app. So you combine with §3 (DO/StepTalk reach in and post) or run from a test binary hosting the app. This is the seed of the in-app agent in §6.</li>
</ul>
<h3>5.2 · External, toolkit-agnostic (works on GNUstep — input only)</h3>
<div class="scroll">
<table class="spec matrix">
<tr><th style="text-align:left">Tool</th><th>Mouse</th><th>Keyboard</th><th>Widget-target</th><th>Read screen</th><th>Runs where</th></tr>
<tr><td><b>XTEST</b> / <b>xdotool</b></td><td class="y">✔</td><td class="y">✔</td><td class="p">window only</td><td class="n">✗</td><td>live X11 / Xvfb</td></tr>
<tr><td>Xnee/cnee (record-replay)</td><td class="y">✔</td><td class="y">✔</td><td class="n">✗</td><td class="n">✗</td><td>live X11 / Xvfb</td></tr>
<tr><td>scrot / import · ImageMagick · tesseract</td><td class="n">✗</td><td class="n">✗</td><td class="p">image/OCR</td><td class="y">✔</td><td>anywhere (files)</td></tr>
<tr><td><b>SikuliX</b> / <b>PyAutoGUI</b> (image-based)</td><td class="y">✔</td><td class="y">✔</td><td class="p">by image</td><td class="y">✔</td><td>live X11 / Xvfb</td></tr>
<tr><td>LDTP / Dogtail / Accerciser (AT-SPI)</td><td class="n">—</td><td class="n">—</td><td class="n">a11y</td><td class="n">a11y</td><td><b>blocked on GNUstep</b></td></tr>
</table>
</div>
<ul class="notes">
<li><b>xdotool</b> (X <b>XTEST</b> extension) is the workhorse: <code class="inl">mousemove</code>/<code class="inl">click</code>/<code class="inl">type</code>/<code class="inl">key</code>. Its one semantic hook is <code class="inl">search --name</code> for top-level windows (GNUstep sets <code class="inl">WM_NAME</code>). <b>Gotcha:</b> window-targeted input uses <code class="inl">XSendEvent</code>, which some GNUstep controls ignore as synthetic — prefer <em>global</em> XTEST injection.</li>
<li class="ok"><b>The per-app CI/local harness:</b> <code class="inl">Xvfb</code> (headless framebuffer) + the app + <code class="inl">xdotool</code> + capture (<code class="inl">scrot</code>) + assert (ImageMagick <code class="inl">%k</code>/<code class="inl">compare</code>, tesseract). <b>SikuliX/PyAutoGUI</b> add "click the thing that looks like this" via OpenCV template-match — the vision stand-in for the missing a11y tree. Add <code class="inl">x11vnc</code> to watch failures. Freeze theme/font/DPI or image/OCR matches drift.</li>
</ul>
<!-- =============== 6 =============== -->
<h2><span class="num">6</span>Acceptance / BDD — a native Gherkin framework (proposed)</h2>
<p class="lead">Human-readable acceptance tests are buildable natively, with direct prior art: <b>Frank</b> and <b>Calabash</b> did this for iOS — Gherkin <code class="inl">.feature</code> files on top, an in-app server that walks the view hierarchy underneath. GNUstep suits the same design, and it <b>routes around the §4 accessibility gap</b>:</p>
<ul class="notes">
<li class="ok"><b>An in-app agent IS the accessibility layer, scoped to testing.</b> Running inside the app, it walks the live <code class="inl">[NSApp windows]</code>→subview tree and reads the real <code class="inl">NSControl</code>/<code class="inl">NSView</code> objects — so "click the button titled OK" and "the field should read X" are <b>native and semantic without AT-SPI</b>. That same view-walking code is the natural seed of the <b>DO-native <code class="inl">NSAccessibility</code> layer (§4.1)</b> — no D-Bus.</li>
</ul>
<div class="flow">
<span class="node">.feature (Gherkin)</span><span class="arr">──▶</span>
<span class="node">godog runner + steps</span><span class="arr">──▶</span>
<span class="node">JSON / Unix socket</span><span class="arr">──▶</span>
<span class="node ok">in-app TestAgent (ObjC/AppKit)</span><span class="arr">──▶</span>
<span class="node">walk view tree · synth NSEvent · read state</span>
</div>
<div class="scroll">
<table class="spec">
<tr><th>Layer</th><th>Choice</th><th>Why</th></tr>
<tr><td>Human layer</td><td>Gherkin <code>.feature</code></td><td>Readable by non-coders; one corpus runs in CI and locally</td></tr>
<tr><td>Runner</td><td><b>godog</b> (Cucumber-for-Go, single static binary) or <b>behave</b> (Python)</td><td>Reused, not built; zero-dep binary is ideal for Actions <em>and</em> local. (No official C Gherkin lib, so a fully-native ObjC runner would embed a parser — reuse is pragmatic.)</td></tr>
<tr><td>Transport</td><td>JSON line protocol over a <b>Unix socket</b></td><td>Language-neutral — the Go/Python runner never speaks ObjC DO</td></tr>
<tr><td>In-app <b>TestAgent</b></td><td>Small GNUstep framework linked into dev/test builds</td><td>Walks the view tree, finds controls by title/tag/class, synthesizes NSEvents (§5.1), reads control state for assertions</td></tr>
<tr><td>Fallback</td><td><code>xdotool</code> + <code>screendump</code>/<code>scrot</code> + OCR</td><td>Same steps drop to pixels for un-instrumented apps or the whole-ISO gate</td></tr>
</table>
</div>
<div class="code"><span class="code-label">features/login.feature — the same file runs in CI and locally</span><pre class="code-body"><span class="k">Feature</span>: First-boot login
<span class="k">Scenario</span>: Admin reaches the desktop
<span class="k">Given</span> the Gershwin session is running
<span class="k">When</span> I log in as <span class="s">"admin"</span> with no password
<span class="k">Then</span> I should see the <span class="s">"System Disk"</span> icon <span class="c"># agent: findControlByTitle != nil (not an OCR guess)</span></pre></div>
<ul class="notes">
<li class="ok"><b>Feasibility verdict:</b> runner is trivial (godog/behave are mature); the <b>TestAgent is the real work but modest</b> — a few hundred lines of ObjC for v1 (buttons, fields, menus, windows + a handful of assert verbs). <b>No accessibility required</b> — that's what makes it feasible today. Start with ~6 verbs and grow. Home it in <code class="inl">gershwin-developer/Library/Tests/</code> as the shared, flavor-agnostic acceptance corpus for all ISOs.</li>
<li class="warn"><b>Honest cost:</b> like Frank/Calabash, apps must link the agent in test/dev builds; where the agent isn't present, the black-box fallback keeps the §4 brittleness. StepTalk (§3) could be an alternate backend for apps shipping a <code class="inl">ScriptingInfo.plist</code>, but a generic view-walking agent works on any GNUstep app unmodified — the better default.</li>
</ul>
<!-- =============== 7 =============== -->
<h2><span class="num">7</span>System / smoke — the current <code style="font-size:0.8em">gershwin-on-nextbsd</code> gate</h2>
<p class="lead">Today's automation is a <b>system/smoke</b> test: boot the ISO in QEMU, drive input through the monitor socket (<b>no guest agent</b>), capture the framebuffer, assert with a colour-count heuristic + OCR. It lives in <code class="inl">.github/workflows/build.yml</code> (<code class="inl">test</code> job) + three <code class="inl">tests/*.sh</code> scripts.</p>
<div class="flow">
<span class="node">boot ISO (QEMU/UEFI, headless)</span><span class="arr">──▶</span>
<span class="node gate">boot-test: greeter painted?</span><span class="arr">──▶</span>
<span class="node">loginwindow-test: sendkey admin</span><span class="arr">──▶</span>
<span class="node gate">workspace-test: desktop up?</span><span class="arr">──▶</span>
<span class="node ok">release continuous</span>
</div>
<div class="code"><span class="code-label">build.yml — QEMU launch (input + capture over one monitor socket)</span><pre class="code-body">qemu-system-x86_64 -machine q35 -m 4G -smp 2 $ACCEL -bios "$OVMF" \
-cdrom "$ISO" -boot d -vga std \
-serial file:tests/serial.log \
-monitor unix:tests/mon.sock,server,nowait \ <span class="c"># sendkey + screendump channel</span>
-nic user,model=e1000 -display none -no-reboot
<span class="c"># ACCEL = "-accel kvm -cpu host" if /dev/kvm else "-accel tcg -cpu qemu64"</span></pre></div>
<div class="code"><span class="code-label">tests/*.sh — inject via socat/sendkey, capture via screendump, assert via %k + OCR</span><pre class="code-body">mon() { printf '%s\n' "$*" | socat -t2 - "UNIX-CONNECT:tests/mon.sock" >/dev/null 2>&1 || true; }
key() { mon "sendkey $1"; sleep 0.2; }
colors() { identify -format '%k' "$1"; } <span class="c"># unique-colour count</span>
<span class="c"># boot-test.sh: PASS when %k > 64 (COLOR_THRESHOLD) → greeter painted</span>
<span class="c"># loginwindow: type "admin", ret, ret (NON-gating, best-effort)</span>
<span class="c"># workspace-test: PASS when %k > 1500 OR tesseract finds "System Disk" (else exit 1)</span></pre></div>
<ul class="notes">
<li><b>Gating:</b> <code class="inl">boot-test</code> and <code class="inl">workspace-test</code> have no <code class="inl">continue-on-error</code> (they fail the job); <code class="inl">loginwindow-test</code> is <code class="inl">continue-on-error: true</code>. The <code class="inl">release</code> job carries <code class="inl">needs: [build, test]</code> + <code class="inl">needs.test.result == 'success'</code>, so a failed desktop check <b>blocks publish</b>. The passing <code class="inl">docs/desktop.png</code> is committed back for the README; <code class="inl">boot-artifacts</code> (serial.log, frames, PPMs) upload on every run.</li>
<li class="ok"><b>What it proves:</b> the ISO boots to a graphical greeter, accepts keystrokes, and reaches a painted desktop. <b>What it does not prove:</b> that any specific widget works — it's a whole-screen heuristic (category: system/smoke), and its "regression" power is limited because it's a colour-count, not a golden-image diff (§9).</li>
</ul>
<!-- =============== 8 =============== -->
<h2><span class="num">8</span>VM-level automation (drives §7; how to add clicks)</h2>
<ul class="notes">
<li><b>Human monitor (today):</b> <code class="inl">sendkey</code>, <code class="inl">mouse_move</code> (relative), <code class="inl">mouse_button</code>, <code class="inl">screendump</code>, over a socket via <code class="inl">socat</code>. Single-client socket — with libvirt use <code class="inl">virsh qemu-monitor-command</code>.</li>
<li class="warn"><b>To add real clicks:</b> monitor <code class="inl">mouse_move</code> is relative (can't hit a pixel reliably). Add <code class="inl">-device usb-tablet</code> and switch clicks to <b>QMP <code class="inl">input-send-event</code></b> (absolute axis + button + key). Assert with <code class="inl">screendump</code> + ImageMagick/tesseract. Stays no-guest-agent — the right layer for whole-ISO tests.</li>
</ul>
<!-- =============== 9 =============== -->
<h2><span class="num">9</span>Regression — cross-cutting</h2>
<p class="lead">Regression isn't a separate tool; it's running the other categories on every change and catching what came back. Native building blocks:</p>
<ul class="notes">
<li class="ok"><b>Unit/integration regression:</b> the GNUstep <b>Testing</b> framework via <code class="inl">make check</code> in CI — exactly how the core libraries guard themselves. Add per-component <code class="inl">Tests/</code> and run on every push.</li>
<li><b>Functional regression:</b> the DO/StepTalk or BDD suites re-run each change.</li>
<li class="warn"><b>Visual regression — the current gap/opportunity:</b> the ISO gate uses a colour-<em>count</em> heuristic, which catches "nothing rendered" but not "the wrong thing rendered." True visual regression = <b>ImageMagick <code class="inl">compare</code></b> a screenshot against a committed golden image (per-region to tolerate clocks/cursors), failing on a metric threshold. Cheap to add on top of the existing <code class="inl">screendump</code> capture, and it upgrades the gate from smoke to real regression.</li>
</ul>
<!-- =============== 10 =============== -->
<h2><span class="num">10</span>Gaps & a full-pyramid strategy</h2>
<div class="pyr">
<div class="row"><div class="bar" style="flex:0 0 62%;border-color:var(--red);color:var(--red)">System / ISO smoke — <b>QEMU gate</b> (have it)</div><div style="flex:1;color:var(--fg-soft)">no guest agent</div></div>
<div class="row"><div class="bar" style="flex:0 0 50%;border-color:var(--amber);color:var(--amber)">Acceptance/UI — <b>BDD + agent</b> (build, §6)</div><div style="flex:1;color:var(--fg-soft)">routes around no-a11y</div></div>
<div class="row"><div class="bar" style="flex:0 0 40%;border-color:var(--accent);color:var(--accent-dark)">Functional/integration — <b>DO / StepTalk</b></div><div style="flex:1;color:var(--fg-soft)">native driver</div></div>
<div class="row"><div class="bar" style="flex:0 0 30%;border-color:var(--green);color:var(--green)">Unit — <b>Testing fw / UnitKit</b></div><div style="flex:1;color:var(--fg-soft)">native, strong</div></div>
</div>
<ul class="notes">
<li class="ok"><b>Bottom of the pyramid is native and ready</b> — adopt the GNUstep Testing framework (<code class="inl">make check</code>) for unit/integration across Gershwin components now; it's the cheapest, most robust win.</li>
<li class="ok"><b>Middle is native driving</b> — use DO/StepTalk for functional/integration of running apps (assert on state, not pixels).</li>
<li class="warn"><b>Top two layers are the gaps</b> — UI, acceptance, and visual regression have no native tooling because of the missing accessibility layer. Fill them with: the <b>in-app TestAgent + Gherkin/BDD</b> (§6, the semantic route), the <b>Xvfb + xdotool + OCR</b> per-app harness (§5.2, the black-box route), and <b>golden-image <code class="inl">compare</code></b> on the ISO gate (§9).</li>
<li><b>Everything runs in GitHub Actions and locally</b> from the same sources: <code class="inl">make check</code> anywhere; godog/Xvfb per-app in seconds (a shared composite action + a local script); the QEMU gate for whole-ISO. Home the shared suites in <code class="inl">gershwin-developer/Library/Tests/</code>.</li>
<li><b>The upstream unlock</b> for first-class UI testing (and assistive tech) is a real <code class="inl">NSAccessibility</code> implementation transported over <b>Distributed Objects</b> (§4.1) — <em>not</em> an AT-SPI2/D-Bus bridge, which is out of scope for Gershwin. It reuses <code class="inl">gdomap</code> as the registry and needs no new bus; the in-app agent is both the near-term substitute and the foundation for it.</li>
</ul>
<!-- =============== 11 =============== -->
<h2><span class="num">11</span>Sources</h2>
<ul class="srcs">
<li>GNUstep Testing framework: <a href="https://wiki.gnustep.org/index.php?title=Implementing_test_cases">Implementing test cases</a> · <a href="https://www.sicpers.info/2013/11/automated-tests-with-the-gnustep-test-framework/">GNUstep test framework write-up</a> · <a href="https://github.com/gnustep/tests-testsuite">tests-testsuite</a></li>
<li><a href="https://github.com/etoile/UnitKit">Étoilé UnitKit</a> · <a href="https://github.com/swiftlang/swift-corelibs-xctest">swift-corelibs-xctest</a></li>
<li><a href="https://www.gnustep.org/experience/StepTalk.html">StepTalk</a> · <a href="https://github.com/gnustep/libs-steptalk">libs-steptalk</a> · <a href="https://mclarenlabs.com/blog/2023/09/22/gnustep-steptalk-is-alive-and-well/">StepTalk is alive & well (2023)</a></li>
<li><a href="https://www.gnustep.org/resources/documentation/Developer/Base/Reference/NSConnection.html">NSConnection / Distributed Objects</a> · <a href="https://github.com/gnustep/libs-base/blob/master/Source/NSConnection.m">NSConnection.m</a></li>
<li><a href="https://github.com/gnustep/libs-gui/blob/master/Source/NSEvent.m">libs-gui NSEvent.m</a> · <a href="https://github.com/gnustep/libs-gui/blob/master/Source/NSAccessibility.m">NSAccessibility.m (stubs)</a></li>
<li><a href="https://raw.githubusercontent.com/gnustep/libs-base/master/Source/NSScriptCommand.m">NSScriptCommand.m (empty stub)</a></li>
<li><a href="https://directory.fsf.org/wiki/At-spi">AT-SPI supported toolkits (GNUstep absent)</a> · <a href="https://www.freedesktop.org/wiki/Accessibility/AT-SPI2/">AT-SPI2 (D-Bus based)</a></li>
<li>DO-native substrate: <a href="https://www.gnustep.org/resources/documentation/Developer/Base/Reference/NSDistributedNotificationCenter.html">NSDistributedNotificationCenter</a> · <code>gdomap</code> (GNUstep DO name server) · macOS <a href="https://developer.apple.com/documentation/applicationservices/axuielement_h">AXUIElement</a> (accessibility without D-Bus)</li>
<li><a href="https://github.com/calabash/calabash-ios">Calabash iOS (Gherkin + embedded server)</a> · Frank (Cucumber + in-app server + UISpec)</li>
<li><a href="https://cucumber.github.io/godog/">godog (Cucumber for Go)</a> · <a href="https://github.com/cucumber/gherkin">cucumber/gherkin parser</a></li>
<li><a href="https://github.com/jordansissel/xdotool">xdotool (XTEST)</a> · <a href="https://man.archlinux.org/man/xdotool.1.en">xdotool man (XTEST vs XSendEvent)</a></li>
<li><a href="https://en.wikipedia.org/wiki/Xvfb">Xvfb</a> · <a href="https://www.jvmhost.com/articles/java-gui-headless-server-xvnc-xvfb-x11vnc/">Xvfb + x11vnc pattern</a></li>
<li><a href="https://imagemagick.org/script/compare.php/">ImageMagick compare (visual regression)</a> · <a href="https://tesseract-ocr.github.io/tessdoc/Command-Line-Usage.html">Tesseract CLI</a></li>
<li><a href="https://sikulix-2014.readthedocs.io/en/latest/devs/system-design.html">SikuliX</a> · <a href="https://pyautogui.readthedocs.io/en/latest/quickstart.html">PyAutoGUI</a> · <a href="https://pypi.org/project/dogtail/">Dogtail (AT-SPI, blocked)</a></li>
<li><a href="https://qemu-project.gitlab.io/qemu/system/monitor.html">QEMU Monitor</a> · <a href="https://en.wikibooks.org/wiki/QEMU/Monitor">sendkey/mouse/screendump</a> · <a href="https://kashyapc.wordpress.com/2013/03/31/multiple-ways-to-access-qemu-machine-protocol-qmp/">QMP input-send-event</a></li>
<li><em>gershwin-on-nextbsd</em> — <code>.github/workflows/build.yml</code>, <code>tests/{boot,loginwindow,workspace}-test.sh</code> (read live from <code>main</code>)</li>
</ul>
<p class="lead dim" style="margin-top:14px;font-size:12.5px"><b>Flagged/unverified:</b> ObjC BDD-unit frameworks (Kiwi/Cedar/Specta) on GNUstep — Apple/XCTest-oriented, compatibility not confirmed. StepTalk 0.10.0 release date (NEWS undated). No existing DO/StepTalk-based GUI-test harness was found (viable, not observed). QEMU <code class="inl">mouse_move</code> under <code class="inl">-device usb-tablet</code> is version-dependent — the reliable absolute-click path is QMP <code class="inl">input-send-event</code>.</p>
<footer>
<p>Prepared for <a href="https://github.com/gershwin-desktop">gershwin-desktop</a> · multi-agent survey of native testing (unit → system) + a live read of the nextbsd screenshot gate · <a href="gershwin-research.html">← Gershwin Research</a></p>
</footer>
</div>
</body>
</html>