Repository navigation
Expand file tree
/
Copy pathframeworks-base.patch
More file actions
4738 lines (4677 loc) · 216 KB
/
Copy pathframeworks-base.patch
File metadata and controls
4738 lines (4677 loc) · 216 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
diff --git a/README.md b/README.md
new file mode 100644
index 000000000000..e169b08cfd3e
--- /dev/null
+++ b/README.md
@@ -0,0 +1,489 @@
+# MobileAgentService — AOSP Framework Modifications
+
+Platform-layer mechanisms for Android that enable GUI agents to operate applications
+efficiently and concurrently with users—without occupying the screen, competing for
+GPU resources, or relying on the screenshot-to-pixel pipeline.
+
+**Branch**: `exp_agent_dirty_readiness_stop` (based on AOSP `25Q3-release`, Android 16)
+
+This branch adds the production dirty/readiness return policy on top of the
+DisplayList export, on-demand traversal, foreground checkpoint, and traversal-only
+freeze branches. Its observation contract is deliberately narrow: after an Agent
+action has been injected, return the first complete active DisplayList for which
+the current UI pipeline is quiet. It does not wait for unrelated future network or
+timer updates to make the page permanently identical.
+
+## Architecture
+
+Three components spanning the windowing system and rendering pipeline:
+
+```
+┌─────────────────────────────────────────────────────────┐
+│ C1: Isolated Agent Task Execution (services/wm) │
+│ ActivityStarter override → same-process dual-instance │
+│ on isolated Virtual Display │
+├─────────────────────────────────────────────────────────┤
+│ C2: DisplayList as UI Representation (hwui) │
+│ RenderNode tree → JSON export via dumpsys gfxinfo │
+├─────────────────────────────────────────────────────────┤
+│ C3: On-Demand Perception, GPU Bypassed (view/hwui) │
+│ Dual-phase traversal: settle (measure+layout only) │
+│ → single record pass → zero GPU via mSyncOnlyFrame │
+│ Per-window freeze + stability detection + profiling │
+└─────────────────────────────────────────────────────────┘
+```
+
+## Modified Files
+
+| File | Purpose |
+|------|---------|
+| `core/java/android/view/ViewRootImpl.java` | Freeze, dual-phase traversal, dirty/readiness observation return, profiling counters; **persistent GPU bypass (`mAgentPersistentNoDrawOverride` tri-state + process-wide `sAgentPersistentNoDrawDefault`, first-frame `mAgentHasDrawnOnce` bootstrap)** |
+| `core/java/android/view/Choreographer.java` | `doFrameForAgent()` synthetic VSYNC pump |
+| `core/java/android/view/ThreadedRenderer.java` | `syncForAgent()` record+sync without GPU draw |
+| `core/java/android/view/WindowManagerGlobal.java` | Parse fresh/nodraw/vsyncN/freeze/traversalfreeze/readysettle/readywaitN and profiling controls; inject agent stats into DL output |
+| `graphics/java/android/graphics/HardwareRenderer.java` | `setSyncOnlyNextFrame()` public hidden API |
+| `libs/hwui/jni/android_graphics_HardwareRenderer.cpp` | JNI bridge for setSyncOnlyNextFrame |
+| `libs/hwui/renderthread/DrawFrameTask.{h,cpp}` | `mSyncOnlyFrame` skips GPU draw after syncFrameState |
+| `libs/hwui/renderthread/RenderProxy.{h,cpp}` | setSyncOnlyNextFrame passthrough |
+| `libs/hwui/pipeline/skia/FlatExportOpsCanvas.h` | C2 export canvas: maps all op bounds to device (screen) space via `getLocalToDeviceAs3x3()`; emits per-run `{font, glyphs}` for external reverse-cmap |
+| `services/core/.../wm/ActivityStarter.java` | C1: detect agent display, override to LAUNCH_MULTIPLE |
+| `services/core/.../wm/ActivityTaskManagerService.java` | `isAgentDisplay()` / mAgentDisplayIds registry |
+| `tools/dlglyph/dl_glyph_decode.py` | Host-side post-processor: reverses glyph IDs back to UTF-8 via fontTools |
+| *(frameworks/native)* `services/surfaceflinger/SurfaceFlinger.{cpp,h}` | **L3 SF compose-bypass: backdoor code 1050 skips compositing an agent VD by layerStack — cuts SF composition GPU + back-pressures video producers without touching window/display/input state (separate repo)** |
+
+## Quick Start: Build & Deploy
+
+```bash
+# Prerequisites: AOSP source tree with this branch checked out at frameworks/base
+cd <AOSP_ROOT>
+source build/envsetup.sh
+lunch gsi_arm64-trunk_staging-userdebug # or your target
+
+# Build (choose based on what you modified):
+m framework-minus-apex -j$(nproc) # core/java + graphics/java changes
+m libhwui -j$(nproc) # libs/hwui C++ changes
+m services -j$(nproc) # services/wm C1 changes
+
+# Deploy to device (must be rooted + remounted):
+SERIAL=<your_device>
+adb -s $SERIAL root && adb -s $SERIAL remount
+
+OUT=out/target/product/<target>/system/framework
+adb -s $SERIAL push $OUT/framework.jar /system/framework/
+adb -s $SERIAL push $OUT/services.jar /system/framework/
+adb -s $SERIAL push out/target/product/<target>/system/lib64/libhwui.so /system/lib64/
+
+# Clear dalvik cache + soft reboot (NEVER hard reboot - overlayfs will lose changes)
+adb -s $SERIAL shell "rm -rf /data/dalvik-cache/*/system@framework@*"
+adb -s $SERIAL shell stop && sleep 3 && adb -s $SERIAL shell start
+```
+
+## API Reference
+
+All interaction through `dumpsys gfxinfo <pkg> displaylist [subparams]`:
+
+| Subparam | Effect |
+|----------|--------|
+| `fresh` | Force on-demand traversal (dual-phase) |
+| `nodraw` | Skip GPU draw (sync-only, zero GPU/SF) |
+| `vsyncN` | Pump up to N synthetic VSYNC frames in Phase 1 (default 3 if no number) |
+| `freeze` | Freeze agent window (stop real-VSYNC self-rendering) |
+| `unfreeze` | Resume normal rendering |
+| `traversalfreeze` | Stop Agent traversal scheduling while preserving the current active DL and renderer content. Use this between Agent steps. |
+| `traversalunfreeze` | Resume Agent traversal scheduling without reconstructing renderer content. |
+| `asyncsettle` | Run settle passes as posted foreground-yielding continuations instead of one synchronous loop. |
+| `readysettle` | Start dirty/readiness settle on the visible Agent root, publish one final active DL, and traversal-freeze atomically. Implies `fresh asyncsettle`. |
+| `readywaitN` | Set the `readysettle` wall-clock fallback to N milliseconds. This is an upper bound, not a minimum sleep. Default: 2000. |
+| `agentstate` | Return compact per-root control/readiness JSON without exporting renderer data. |
+| `gpubypass` | Persistent no-draw: EVERY traversal (incl. the app's own real-VSYNC frames while unfrozen) skips the GPU draw. Process-wide default, inherited by newly-navigated agent windows automatically. Agent-UI only. Independent of freeze. |
+| `nogpubypass` | Disable persistent no-draw (resume real drawing) |
+| `reset` | Reset per-window profiling counters |
+
+### Output Format
+
+Per-window header with agent stats followed by DL JSON:
+```
+<pkg/activity/VRI@hash> (visibility=N){"settled":B,"phase1_traversals":P1,"phase2_traversals":P2}{"frame":"flat_render","width":W,"height":H,"nodes":[...]}
+```
+
+- `settled=true`: Phase 1 exited via early-break (view tree stable)
+- `settled=false`: Phase 1 exhausted vsyncN budget (perpetual animation present)
+- `phase1_traversals`: cumulative measure+layout-only passes since last reset
+- `phase2_traversals`: cumulative full-record passes since last reset
+- `observe_result=readiness`: dirty/readiness ended the observation before the fallback
+- `observe_result=timeout`: `readywaitN` fired and the fallback snapshot was published
+- `observe_result=sync_failed|detached`: no consumable snapshot; the caller must retry or use pixel fallback
+- `observe_duration_ms` / `observe_passes`: device-side return latency and settle passes
+- `last_sync_valid=true,last_sync_result=0`: the final root DL is valid and crossed the
+ RenderThread staging-to-active sync boundary
+
+When a package has both physical-display and Agent-display roots, match the root whose
+stats contain `"agent_ui":true`, the expected `"display":<VD_ID>`, and
+`visibility=0`. Do not select a window by package name or list order alone.
+
+Each op's `bounds`/`dst` is in **device (screen) coordinates** — the export
+canvas maps every local-space input through `SkCanvas::getLocalToDeviceAs3x3()`
+at the time of capture, so rectangles can be used directly as tap targets
+without further math.
+
+`drawText` ops additionally carry the SkTextBlob's raw runs:
+
+```json
+{"op":"drawText","x":221,"y":309,"bounds":[224,322,565,355],"size":33,
+ "color":"#ff30323b",
+ "runs":[{"font":"Roboto","glyphs":[51,74,89,92,84,87,80,5,11,5,78,83,89,74,87,83,74,89]}]}
+```
+
+The framework intentionally does NOT do glyph→UTF-8 reverse-cmap inline. That
+step is a pure font-table lookup; doing it in C++ would require shipping
+GSUB/cmap logic and font fallback into HWUI for no benefit. Instead, the host
+script `tools/dlglyph/dl_glyph_decode.py` performs the reverse with `fontTools`
+(see *DisplayList Usability* section below).
+
+## Usage Examples
+
+### Basic: Get fresh DL with GPU bypassed
+```bash
+adb shell "dumpsys gfxinfo com.example.app displaylist fresh nodraw"
+```
+
+### With stability detection (wait up to 2s = 120 frames @60Hz)
+```bash
+adb shell "dumpsys gfxinfo com.example.app displaylist fresh nodraw vsync120"
+```
+
+### Recommended workflow: action to readiness-aligned observation
+```bash
+# 1. Start user instance on display 0
+adb shell am start --display 0 -n com.example.app/.MainActivity
+
+# 2. Create agent Virtual Display and launch agent instance
+app_process -cp /data/local/tmp/vd_agent.dex / \
+ com.agent.VirtualDisplayAgent offdump com.example.app/.MainActivity 600 iso
+
+# 3. Get agent display ID from log
+DID=$(cat /data/local/tmp/vd_off.log | grep -oE "READY displayId=[0-9]+" | grep -oE "[0-9]+")
+
+# 4. Let the new Agent window complete its first real draw, then preserve that
+# active DL while stopping only traversal scheduling between steps.
+adb shell "dumpsys gfxinfo com.example.app displaylist agentctl traversalfreeze"
+
+# 5. Inject one complete action on the Agent display. The command must return
+# before starting observation; DOWN-only injection is not an action boundary.
+adb shell input -d $DID tap 540 800
+
+# 6. Immediately request the post-action observation. Dirty/readiness may return
+# early; 2000 ms is the former fixed wait retained only as a fallback bound.
+adb exec-out "dumpsys gfxinfo com.example.app displaylist fresh nodraw \
+ asyncsettle readysettle readywait2000" > /tmp/post_action_dl.txt
+
+# 7. Verify the selected Agent root before giving the DL to the next Agent step.
+adb shell "dumpsys gfxinfo com.example.app displaylist agentstate"
+# Accept: observe_result is readiness or timeout, frozen=true,
+# last_sync_valid=true, last_sync_result=0, observe_active=false.
+# Reject: sync_failed, detached, wrong display/root, empty DL, ANR/login/ad UI.
+```
+
+`readysettle` unfreezes the visible Agent root internally. After a stable candidate,
+it performs one complete Java DL record and synchronous RenderThread handoff, checks
+that recording did not schedule more UI work, and then traversal-freezes before the
+dumpsys output is generated. If readiness never appears, the same final record/sync
+runs at `readywaitN`. The command therefore replaces both the old unconditional
+post-action sleep and the additional sleep used to wait for an unfrozen traversal.
+
+## Caller Alignment
+
+The readiness request must be aligned to an Agent action, not to inference time or a
+host polling interval:
+
+| Old caller behavior | This branch |
+|---|---|
+| `sleep(wait_after_action_seconds)` | Remove the sleep; call `readysettle` immediately after action injection returns |
+| Unfreeze, then `sleep(300 ms)` before dumping DL | Remove both; `readysettle` owns unfreeze, settle, final sync, and refreeze |
+| `wait_after_action_seconds=2.0` | `readywait2000`; preserve the value as fallback only |
+| `freeze` between every step | `traversalfreeze`; avoid per-step `clearContent()` and DL reconstruction |
+| Assume the first package window is the Agent UI | Select `agent_ui=true`, matching VD display ID, `visibility=0` |
+| Treat every dumpsys return as valid | Gate on result, sync validity, root/display, non-empty DL, and blocking UI |
+
+For this workspace, the aligned commercial caller is
+`agent_os/experiments/exp_commercial_three_configs_profiled/code/m3a_dl_agent.py`.
+It records before/after `observe_result`, `observe_duration_ms`, `observe_passes`, and
+Phase-1/2 counters in each `step.json`. The device-side smoke entry is:
+
+```bash
+cd /home/lishengkai/scripts/agent_os/experiments
+python3 exp_displaylist_ondemand/smoke_ready_settle.py \
+ --serial <serial> --adb-port <port> --package <pkg> --display-id <VD_ID> \
+ --action swipe_up --action tap:540:1200 --ready-wait-ms 2000
+```
+
+The smoke tool assumes the Agent VD and target app are already running. It does not
+create, reassign, or destroy displays, which makes it safe to hand to a tester after
+the device owner has confirmed the display and package.
+
+### Profiling: measure traversal counts and GPU frames
+```bash
+# Reset counters at task start
+adb shell "dumpsys gfxinfo com.example.app displaylist reset"
+
+# ... run agent task (multiple actions + observations) ...
+
+# Check cumulative stats
+adb shell "dumpsys gfxinfo com.example.app displaylist fresh nodraw" | \
+ grep -oP '(?<=visibility=\d\))\{[^{]*\}'
+# Output: {"settled":true,"phase1_traversals":5,"phase2_traversals":3}
+
+# Compare with GPU frames (should be 0 for agent window while frozen)
+adb shell "dumpsys gfxinfo com.example.app" | grep "Total frames rendered"
+```
+
+## DisplayList Usability
+
+This section quantifies what the DL representation actually delivers as an
+agent-side input, relative to screenshots and the AccessibilityService XML.
+
+### What the export captures
+
+Each `drawText`/`drawRect`/`drawPath`/`drawImage`/... op in the JSON carries:
+- **Device-space bounds** — directly usable as tap targets, no matrix math
+ on the consumer side
+- **Color** (ARGB hex), **font size**, **stroke style** where applicable
+- For text: **font family name** + **raw glyph ID array** per Skia run
+- For nested RenderNodes: hierarchical `drawDrawable.ops[...]` blocks that
+ preserve the rendering tree (incl. HardwareLayers)
+
+Three things the export does NOT carry, by design:
+1. **Image pixel content** — `drawImage` emits dst rect + source w×h only.
+ This is the only inherent limit of the DisplayList channel; agents that
+ need image semantics still need a vision model for that subregion.
+2. **View identity / clickable flags** — the op stream is a rendering view;
+ `View.isClickable()` is application semantics and belongs in a thin
+ side-channel (~50 bytes per interactive View), not in the op stream itself.
+3. **UTF-8 text** — only glyph IDs are emitted (Skia loses the source text
+ once it shapes into a TextBlob). Recovered by the host-side script below.
+
+### Glyph → UTF-8 recovery (deterministic, model-free)
+
+`tools/dlglyph/dl_glyph_decode.py` walks each `runs[]` entry and reverse-maps
+glyph IDs to Unicode codepoints via `fontTools.ttLib`. For TrueType
+Collections (Noto CJK ships as a TTC of 5 region variants), the cmaps of all
+sub-fonts are unioned under the same family name.
+
+```bash
+# One-time: pull device fonts
+adb -s <serial> pull /system/fonts/Roboto-Regular.ttf ~/fonts/
+adb -s <serial> pull /system/fonts/NotoSansCJK-Regular.ttc ~/fonts/
+
+# Optional: pull app-private fonts (rarely needed)
+adb -s <serial> shell pm path <pkg> | sed 's/^package://' | \
+ xargs -I {} adb -s <serial> pull {} /tmp/app.apk
+unzip -j /tmp/app.apk "assets/*.ttf" "assets/*.otf" -d ~/fonts/
+
+# Capture + decode
+adb -s <serial> exec-out "dumpsys gfxinfo <pkg> displaylist fresh nodraw" \
+ > /tmp/dl.json
+tools/dlglyph/dl_glyph_decode.py --fonts ~/fonts /tmp/dl.json \
+ -o /tmp/dl_decoded.json
+```
+
+The decoded JSON has the same structure as the raw output, with each
+`drawText` op gaining a `text` field:
+
+```json
+{"op":"drawText","bounds":[224,322,565,355],"size":33,
+ "color":"#ff30323b","text":"Network & internet", ...}
+```
+
+### Empirical recovery rate
+
+Measured on Settings home, Zhihu home, Bilibili home (Android 16 GSI, Pixel 10
+Pro) — every `drawText` op extracted, every glyph in every run reverse-mapped:
+
+| App | Total glyphs | Recovered | Rate |
+|----------|-------------:|----------:|--------:|
+| Settings | 735 | 735 | 100.0% |
+| Zhihu | 990 | 990 | 100.0% |
+| Bilibili | 288 | 287 | 99.7% |
+
+By font:
+
+| Font | Glyphs | Recovered | Rate |
+|-------------------------|-------:|----------:|--------:|
+| Roboto (Latin/digits) | 1,519 | 1,519 | 100.0% |
+| Noto Sans CJK SC | 493 | 493 | 100.0% |
+| App-private iconfont | 1 | 0 | 0.0% |
+
+The single missed glyph on Bilibili comes from an in-app font name reported
+by `SkTypeface::getFamilyName()` as `"bilibili"`. The corresponding glyph is
+not present in any of the system fonts; if the app's `assets/*.ttf` is
+unpacked and supplied via `--fonts`, that gap closes too.
+
+### Known limitations (and what each costs)
+
+| Source of loss | Effect | Severity for Chinese/English UI |
+|----------------|--------|---------------------------------|
+| App-private fonts not unpacked from APK | Glyphs in that font reverse to `�` | 0.05% on the test set; fix is one-time per app |
+| GSUB ligatures (Latin `fi`, `ffi`) | Reverse yields one codepoint instead of two | Negligible — apps rarely use Latin ligatures |
+| Complex shaping (Arabic, Devanagari) | Glyph order ≠ logical char order; positional variants need GSUB reverse | Out of scope for our target apps |
+| Unicode variant codepoints (e.g. Kangxi radical "⻔" vs "门") | Decoded char is visually identical but has a different codepoint | Visual equivalence preserved; downstream string match needs normalization |
+| Image content (`drawImage`) | Only dst-rect + w/h, no pixels | Inherent to DL channel; out of scope |
+
+### What this means relative to existing channels
+
+- **vs Screenshot**: the DL op stream carries equivalent visual information
+ (modulo image internals) in structured form — a button rendered as
+ drawRRect-background + centered drawText is recognizable from the op
+ attributes the same way a VLM recognizes it from pixels, but at a fraction
+ of the token cost.
+- **vs AccessibilityService**: DL captures every glyph that is drawn,
+ including text rendered by custom `canvas.drawText` calls that the a11y
+ layer cannot see. Conversely, DL does not carry `clickable`/`focusable`
+ flags — those come from `View` state and require a thin side-channel.
+- **Limits that remain after this work**: only image pixel content (the one
+ inherent limit) and the optional `{id, clickable, role}` side-channel
+ (~50 bytes/View) the agent needs to disambiguate semantically-clickable-
+ but-visually-passive elements.
+
+## Experiment Notes
+
+1. **Never `adb reboot`** — overlayfs is tmpfs, hard reboot loses all pushed changes.
+ Always use `adb shell stop && start` for soft reboot.
+
+2. **Never `m framework`** — that builds the class jar, not the DEX jar.
+ Always use `m framework-minus-apex`.
+
+3. **C1 verification**: check logcat for `AgentDisplay: Overriding to LAUNCH_MULTIPLE`
+ after launching on agent display.
+
+4. **GPU isolation verification**: after traversal-freeze, `Total frames rendered` should not
+ increase for the agent window. Small increments (1-4) may come from system UI on
+ display 0, not from the agent window.
+
+5. **`uiautomator dump --display N`** is unreliable in multi-display setups — always
+ use `dumpsys gfxinfo ... displaylist` + `screencap` for cross-validation.
+
+6. **Display ID is dynamic** — each `vd_agent` launch gets a new ID. Always read it
+ from the vd_agent log, never hardcode.
+
+7. **Readiness fallback selection**:
+ - Start validation with `readywait2000` to align with the previous commercial
+ runner's `wait_after_action_seconds=2.0`.
+ - Keep `readywaitN` constant across compared configurations; report readiness and
+ timeout samples separately.
+ - `readywaitN` does not certify future network completion. It bounds when the
+ current action observation is delivered.
+ - `vsyncN` remains a legacy pass-budget interface for synchronous experiments; do
+ not mix it with production readiness latency numbers.
+
+## Commit History
+
+Current branch `exp_agent_dirty_readiness_stop` (newest first):
+```
+6b300f7a Agent UI: stop observations on cheap readiness
+de2203f8 Agent UI: refine cheap readiness profiling
+654664c6 Agent UI: profile DisplayList readiness generations
+daa61196 Agent UI: add traversal-only freeze controls
+fa317639 Agent UI: checkpoint on-demand traversals
+2eef669a Agent UI: dispatch foreground frames at node checkpoints
+8e6fb21d Agent C3 L2: persistent GPU bypass as an auto agent-UI window property
+6ed448b dlglyph: multi-candidate font matching + CJK word-freq disambiguation
+1bb67aae Agent C3: release agent window GPU resources on freeze (agent-UI-only)
+1005f63e Agent C3 Layer-1: idle-driven decouple traversal + anti-starvation bound
+e4b6f442 add bounds and drawimage/drawpath functions (DL visual completion)
+a90aa32d Agent C3 Layer-1: make decouple an automatic window property of agent UIs
+cb00a319 Agent C3: three-layer decoupled architecture (callback reroute / GPU bypass / DL semantic-hash stability)
+42f0934d Agent on-demand: add display id + frozen flag to stats JSON
+d58973d5 Agent on-demand: switch settle to posted self-continuation
+c8fcd683 Agent on-demand: idle-driven foreground-yielding settle + animation attribution
+831e8169 DisplayList Export: device-space op bounds + per-run glyph IDs + reverse-cmap script
+```
+
+### Three-layer GPU-bypass design (current)
+
+The agent UI's GPU cost is cut by three orthogonal, independently-switchable,
+agent-UI-only layers. None touches window visibility / display power / input /
+activity lifecycle, so the agent stays perceivable (DisplayList) and operable (tap).
+The table separately lists the long-idle resource-release variant of L1 because it
+must not be substituted for the per-step scheduling freeze:
+
+| Layer | Cuts | Mechanism | Switch |
+|-------|------|-----------|--------|
+| **L1 scheduling freeze** | no Agent traversal between steps while retaining the active DL | `setAgentTraversalFrozen`: `scheduleTraversals` early-return, no `clearContent()` | `displaylist traversalfreeze` / `traversalunfreeze`; `readysettle` refreezes automatically |
+| **Long-idle resource freeze** | additionally releases this window's GPU resources when reconstruction cost is acceptable | `setAgentFrozen`: scheduling gate + `clearContent()` | `displaylist freeze` / `unfreeze`; do not use per Agent step |
+| **L2 persistent no-draw** | EVERY traversal skips `context->draw()` (rasterization + SF submit), incl. the app's own VSYNC frames — not just the observe frame | `mAgentPersistentNoDraw*`: `performDraw` routes to `syncForAgent` (record DL + RT sync, skip draw). Process-wide default so navigated sub-windows inherit; first real frame builds the DisplayList, then bypass | `displaylist gpubypass` / `nogpubypass` |
+| **L3 SF compose-bypass** | SF's composition pass for the agent VD + video/SurfaceView producer back-pressure | *(frameworks/native)* SurfaceFlinger backdoor code 1050 skips the agent VD's Output by layerStack | `service call SurfaceFlinger 1050 i32 <displayId> i32 <1/0>` |
+
+Measured (2-round avg, c3_dl_nofg_compose vs bgui_nofg baseline): target-app GPU
+compute **−94.6%**, SF composition GPU **−95.1%**, rendered frames ≈ **−99%**
+(agent VD renders single-digit frames vs thousands). CPU is roughly flat per unit
+time (DL export + agent scheduling replace the rasterization CPU). WebView pages
+(Firefox: GPU in a separate `:gpu` process) are an intrinsic boundary the render-
+pipeline bypass cannot reach.
+
+## License
+
+Same as AOSP — Apache 2.0.
+
+## C2 DisplayList Export 修改说明
+
+本分支在 Android `frameworks/base` 中加入了 C2 使用的 DisplayList 导出能力。该修改的目标是从 Android RenderThread / HWUI 渲染流程中导出当前窗口的绘制指令,并将其转换为结构化 JSON,供上层 agent 使用。
+
+### 修改目标
+
+原始 Android framework 只能通过截图或 ViewTree 获取界面信息。为了让 agent 能够直接访问 UI 的底层绘制结构,本修改在 HWUI 中加入了 DisplayList 导出逻辑,使系统可以输出如下信息:
+
+- 当前窗口的绘制操作,例如 `drawText`、`drawPath`、`drawImage`、`drawRect` 等;
+- 文本内容、绘制 bounds、颜色、图片采样信息;
+- 图标和 path 的几何形状;
+- 元素在全局屏幕坐标系中的位置;
+- 当前窗口的 surface origin 和 surface insets 信息。
+
+这些信息会被后续 Python 侧脚本进一步过滤、简化,并转换为 agent observation。
+
+### 核心修改
+
+本次修改涉及 Java framework、JNI、RenderProxy、CanvasContext 和 HWUI Skia pipeline。主要修改文件如下:
+
+```text
+core/java/android/view/ThreadedRenderer.java
+core/java/android/view/WindowManagerGlobal.java
+graphics/java/android/graphics/HardwareRenderer.java
+libs/hwui/jni/android_graphics_HardwareRenderer.cpp
+libs/hwui/pipeline/skia/FlatExportOpsCanvas.h
+libs/hwui/renderthread/CanvasContext.cpp
+libs/hwui/renderthread/CanvasContext.h
+libs/hwui/renderthread/RenderProxy.cpp
+libs/hwui/renderthread/RenderProxy.h
+```
+
+### FlatExportOpsCanvas
+
+`FlatExportOpsCanvas` 是本修改的核心组件。它继承自 `SkCanvas`,在 DisplayList replay 过程中拦截绘制操作,并导出 JSON 格式的绘制记录。
+
+目前支持导出的主要绘制操作包括:
+
+- `drawTextBlob`
+- `drawPath`
+- `drawImage`
+- `drawRect`
+- `drawOval`
+- `drawRRect`
+- `drawCircle`
+- `clipRect`
+
+对于不同操作,会导出对应的结构化信息,例如:
+
+- `op`
+- `bounds`
+- `color`
+- `text`
+- `path`
+- `src`
+- `dst`
+- `imageId`
+- `visualStatus`
+- `sampleWidth`
+- `sampleHeight`
+- `argbRle`
+
+其中 `drawImage` 会尝试导出小尺寸 ARGB 采样,并使用 run-length encoding 表示图片像素信息,供后续脚本生成图标语义描述。
diff --git a/core/java/android/view/Choreographer.java b/core/java/android/view/Choreographer.java
index 3a2ec91b3b20..fcf46feca294 100644
--- a/core/java/android/view/Choreographer.java
+++ b/core/java/android/view/Choreographer.java
@@ -32,6 +32,7 @@ import android.graphics.Insets;
import android.hardware.display.DisplayManagerGlobal;
import android.os.Build;
import android.os.Handler;
+import android.os.HandlerThread;
import android.os.Looper;
import android.os.Message;
import android.os.SystemClock;
@@ -41,6 +42,8 @@ import android.util.Log;
import android.util.TimeUtils;
import android.view.animation.AnimationUtils;
+import com.android.internal.annotations.GuardedBy;
+
import java.io.PrintWriter;
import java.util.Locale;
import java.util.concurrent.atomic.AtomicBoolean;
@@ -353,6 +356,31 @@ public final class Choreographer {
setFPSDivisor(SystemProperties.getInt(ThreadedRenderer.DEBUG_FPS_DIVISOR, 1));
}
+ // Agent node-checkpoint prototype. A receiver on the main Looper cannot publish VSYNC
+ // while an agent traversal occupies that Looper, so a side receiver mirrors requested
+ // app VSYNCs into this mailbox. Only the main thread consumes the mailbox.
+ private HandlerThread mAgentCheckpointVsyncThread;
+ private AgentCheckpointDisplayEventReceiver mAgentCheckpointDisplayEventReceiver;
+ @GuardedBy("mLock")
+ private boolean mAgentCheckpointTraversalActive;
+ @GuardedBy("mLock")
+ private boolean mAgentCheckpointVsyncScheduled;
+ @GuardedBy("mLock")
+ private boolean mAgentCheckpointVsyncPending;
+ @GuardedBy("mLock")
+ private boolean mAgentCheckpointFrameDispatching;
+ @GuardedBy("mLock")
+ private long mAgentCheckpointVsyncTimestampNanos;
+ @GuardedBy("mLock")
+ private int mAgentCheckpointVsyncFrame;
+ @GuardedBy("mLock")
+ private final DisplayEventReceiver.VsyncEventData mAgentCheckpointVsyncEventData =
+ new DisplayEventReceiver.VsyncEventData();
+ @GuardedBy("mLock")
+ private long mLastAgentCheckpointVsyncId = FrameInfo.INVALID_VSYNC_ID;
+ @GuardedBy("mLock")
+ private long mLastAgentCheckpointVsyncTimestampNanos = Long.MIN_VALUE;
+
private static float getRefreshRate() {
DisplayInfo di = DisplayManagerGlobal.getInstance().getDisplayInfo(
Display.DEFAULT_DISPLAY);
@@ -419,6 +447,22 @@ public final class Choreographer {
private void dispose() {
mDisplayEventReceiver.dispose();
+ final AgentCheckpointDisplayEventReceiver checkpointReceiver;
+ final HandlerThread checkpointThread;
+ synchronized (mLock) {
+ checkpointReceiver = mAgentCheckpointDisplayEventReceiver;
+ checkpointThread = mAgentCheckpointVsyncThread;
+ mAgentCheckpointDisplayEventReceiver = null;
+ mAgentCheckpointVsyncThread = null;
+ mAgentCheckpointTraversalActive = false;
+ mAgentCheckpointVsyncPending = false;
+ }
+ if (checkpointReceiver != null) {
+ checkpointReceiver.dispose();
+ }
+ if (checkpointThread != null) {
+ checkpointThread.quitSafely();
+ }
}
/**
@@ -1191,6 +1235,129 @@ public final class Choreographer {
}
}
+ /**
+ * Synthetic VSYNC pump for agent on-demand traversal. Runs the full doFrame
+ * callback chain (INPUT/ANIMATION/INSETS_ANIMATION/TRAVERSAL/COMMIT) on the UI
+ * thread without waiting for a real display VSYNC. This advances animations,
+ * frame callbacks and content (image decode commits, lazy-load) so the recorded
+ * DisplayList reflects settled content even while the display is STATE_OFF.
+ * Must be called on this Choreographer's looper thread.
+ * @hide
+ */
+ public void doFrameForAgent() {
+ final long now = System.nanoTime();
+ synchronized (mLock) {
+ // doFrame early-returns "no work to do" unless a frame is scheduled.
+ mFrameScheduled = true;
+ }
+ DisplayEventReceiver.VsyncEventData data = new DisplayEventReceiver.VsyncEventData();
+ // ~60Hz interval; a real value (not -1) so jitter/timeline logic stays sane.
+ data.frameInterval = 16_666_666L;
+ doFrame(now, 0, data);
+ }
+
+ /**
+ * Whether a frame is currently scheduled on this (shared, per-thread) Choreographer.
+ * Used by the agent on-demand idle-driven settle to yield to the foreground: if a
+ * foreground frame is pending, the agent defers its next settle pass.
+ * @hide
+ */
+ public boolean isFrameScheduledForAgentCheck() {
+ synchronized (mLock) {
+ return mFrameScheduled;
+ }
+ }
+
+ /** Begin an agent traversal that may service an arrived foreground VSYNC inline. @hide */
+ public void beginAgentNodeCheckpointTraversal() {
+ ensureAgentCheckpointDisplayEventReceiver();
+ synchronized (mLock) {
+ mAgentCheckpointTraversalActive = true;
+ mAgentCheckpointVsyncPending = false;
+ // Keep the side receiver subscribed while the agent owns the main Looper. A frame
+ // request posted from another thread cannot run MSG_DO_SCHEDULE_VSYNC until the
+ // traversal returns, but it does set mFrameScheduled under mLock immediately.
+ scheduleAgentCheckpointVsyncLocked();
+ }
+ }
+
+ /** End an agent node-checkpoint traversal and restore the vanilla VSYNC path. @hide */
+ public void endAgentNodeCheckpointTraversal() {
+ synchronized (mLock) {
+ mAgentCheckpointTraversalActive = false;
+ mAgentCheckpointVsyncPending = false;
+ if (mFrameScheduled && mDisplayEventReceiver != null) {
+ mDisplayEventReceiver.scheduleVsync();
+ }
+ }
+ }
+
+ /**
+ * Runs one already-arrived foreground frame while preserving the caller's agent traversal
+ * stack. This is deliberately a frame-only pump; it does not dispatch arbitrary Looper work.
+ * @hide
+ */
+ public boolean doFrameAtAgentNodeCheckpoint() {
+ final long timestampNanos;
+ final int frame;
+ final DisplayEventReceiver.VsyncEventData data =
+ new DisplayEventReceiver.VsyncEventData();
+ synchronized (mLock) {
+ if (!mAgentCheckpointTraversalActive || mAgentCheckpointFrameDispatching
+ || !mAgentCheckpointVsyncPending || !mFrameScheduled) {
+ return false;
+ }
+ mAgentCheckpointVsyncPending = false;
+ mAgentCheckpointFrameDispatching = true;
+ timestampNanos = mAgentCheckpointVsyncTimestampNanos;
+ frame = mAgentCheckpointVsyncFrame;
+ data.copyFrom(mAgentCheckpointVsyncEventData);
+ mLastAgentCheckpointVsyncId = data.preferredFrameTimeline().vsyncId;
+ mLastAgentCheckpointVsyncTimestampNanos = timestampNanos;
+ }
+ Trace.traceBegin(Trace.TRACE_TAG_VIEW, "agentNodeCheckpoint#doFrame");
+ try {
+ doFrame(timestampNanos, frame, data);
+ return true;
+ } finally {
+ synchronized (mLock) {
+ mAgentCheckpointFrameDispatching = false;
+ }
+ Trace.traceEnd(Trace.TRACE_TAG_VIEW);
+ }
+ }
+
+ private void ensureAgentCheckpointDisplayEventReceiver() {
+ synchronized (mLock) {
+ if (mAgentCheckpointDisplayEventReceiver != null || !USE_VSYNC) {
+ return;
+ }
+ }
+ final HandlerThread thread = new HandlerThread("AgentCheckpointVsync");
+ thread.start();
+ final AgentCheckpointDisplayEventReceiver receiver =
+ new AgentCheckpointDisplayEventReceiver(thread.getLooper());
+ synchronized (mLock) {
+ if (mAgentCheckpointDisplayEventReceiver == null) {
+ mAgentCheckpointVsyncThread = thread;
+ mAgentCheckpointDisplayEventReceiver = receiver;
+ return;
+ }
+ }
+ receiver.dispose();
+ thread.quitSafely();
+ }
+
+ @GuardedBy("mLock")
+ private void scheduleAgentCheckpointVsyncLocked() {
+ if (!mAgentCheckpointTraversalActive || mAgentCheckpointVsyncScheduled
+ || mAgentCheckpointDisplayEventReceiver == null) {
+ return;
+ }
+ mAgentCheckpointVsyncScheduled = true;
+ mAgentCheckpointDisplayEventReceiver.scheduleVsync();
+ }
+
void doScheduleVsync() {
synchronized (mLock) {
if (mFrameScheduled) {
@@ -1214,7 +1381,12 @@ public final class Choreographer {
private void scheduleVsyncLocked() {
try {
Trace.traceBegin(Trace.TRACE_TAG_VIEW, "Choreographer#scheduleVsyncLocked");
- mDisplayEventReceiver.scheduleVsync();
+ if (mAgentCheckpointTraversalActive
+ && mAgentCheckpointDisplayEventReceiver != null) {
+ scheduleAgentCheckpointVsyncLocked();
+ } else {
+ mDisplayEventReceiver.scheduleVsync();
+ }
} finally {
Trace.traceEnd(Trace.TRACE_TAG_VIEW);
}
@@ -1550,10 +1722,48 @@ public final class Choreographer {
@Override
public void run() {
mHavePendingVsync = false;
+ synchronized (mLock) {
+ final long vsyncId = mLastVsyncEventData.preferredFrameTimeline().vsyncId;
+ final boolean consumedAtAgentCheckpoint =
+ (vsyncId != FrameInfo.INVALID_VSYNC_ID
+ && vsyncId <= mLastAgentCheckpointVsyncId)
+ || mTimestampNanos <= mLastAgentCheckpointVsyncTimestampNanos;
+ if (consumedAtAgentCheckpoint) {
+ if (mFrameScheduled && !mAgentCheckpointTraversalActive) {
+ mDisplayEventReceiver.scheduleVsync();
+ }
+ return;
+ }
+ }
doFrame(mTimestampNanos, mFrame, mLastVsyncEventData);
}
}
+ private final class AgentCheckpointDisplayEventReceiver extends DisplayEventReceiver {
+ AgentCheckpointDisplayEventReceiver(Looper looper) {
+ super(looper, VSYNC_SOURCE_APP, /* eventRegistration */ 0, /* layerHandle */ 0L);
+ }
+
+ @Override
+ public void onVsync(long timestampNanos, long physicalDisplayId, int frame,
+ VsyncEventData vsyncEventData) {
+ synchronized (mLock) {
+ mAgentCheckpointVsyncScheduled = false;
+ if (!mAgentCheckpointTraversalActive) {
+ return;
+ }
+ if (mFrameScheduled) {
+ mAgentCheckpointVsyncTimestampNanos =
+ Math.min(timestampNanos, System.nanoTime());
+ mAgentCheckpointVsyncFrame = frame;
+ mAgentCheckpointVsyncEventData.copyFrom(vsyncEventData);
+ mAgentCheckpointVsyncPending = true;
+ }
+ scheduleAgentCheckpointVsyncLocked();
+ }
+ }
+ }
+
private static final class CallbackRecord {
public CallbackRecord next;
public long dueTime;
diff --git a/core/java/android/view/ThreadedRenderer.java b/core/java/android/view/ThreadedRenderer.java
index ad46f2b6df84..e04f3e43d4f8 100644
--- a/core/java/android/view/ThreadedRenderer.java
+++ b/core/java/android/view/ThreadedRenderer.java
@@ -661,15 +661,33 @@ public final class ThreadedRenderer extends HardwareRenderer {
/** @hide */
public static void handleDumpGfxInfo(FileDescriptor fd, String[] args) {
- dumpGlobalProfileInfo(fd, dumpArgsToFlags(args));
+ boolean agentCommand = false;
+ for (String arg : args) {
+ if ("agentctl".equals(arg) || "agentstate".equals(arg)) {
+ agentCommand = true;
+ break;
+ }
+ }
+ if (!agentCommand) {
+ dumpGlobalProfileInfo(fd, dumpArgsToFlags(args));
+ }
WindowManagerGlobal.getInstance().dumpGfxInfo(fd, args);
}
/**
* Outputs extra debugging information in the specified file descriptor.
*/
- void dumpGfxInfo(PrintWriter pw, FileDescriptor fd, String[] args) {
+ void dumpGfxInfo(PrintWriter pw, FileDescriptor fd, String[] args,
+ int surfaceOriginX, int surfaceOriginY, int surfaceInsetX, int surfaceInsetY) {
pw.flush();
+ // Check if displaylist dump is requested: 'dumpsys gfxinfo <pkg> displaylist'
+ for (String arg : args) {
+ if ("displaylist".equals(arg)) {
+ dumpDisplayList(fd, surfaceOriginX, surfaceOriginY,
+ surfaceInsetX, surfaceInsetY);
+ return;
+ }
+ }
dumpProfileInfo(fd, dumpArgsToFlags(args));
}
@@ -792,6 +810,48 @@ public final class ThreadedRenderer extends HardwareRenderer {
* @param view The view to draw.
* @param attachInfo AttachInfo tied to the specified view.
*/
+ /**
+ * Agent no-draw sync: record the View tree's DisplayList and sync it to the
+ * RenderThread (staging->active) but skip the GPU draw. Mirrors {@link #draw}
+ * minus the actual frame draw, giving an up-to-date DisplayList with zero
+ * GPU/SurfaceFlinger work.
+ */
+ void syncForAgent(View view, AttachInfo attachInfo, DrawCallbacks callbacks) {
+ final ViewRootImpl root = attachInfo.mViewRootImpl;
+ final boolean profileDlReadiness = root.isAgentDlProfileRecordingActive();
+ root.mViewFrameInfo.markDrawStart();
+ if (profileDlReadiness) {
+ root.agentDlProfileOnRecordPassBegin();
+ }
+ final long recordStartNanos = profileDlReadiness ? System.nanoTime() : 0;
+ updateRootDisplayList(view, callbacks);
+ if (profileDlReadiness) {
+ root.agentDlProfileOnRecordPassComplete(
+ mRootNode.hasDisplayList(), view.getUniqueDrawingId(),
+ System.nanoTime() - recordStartNanos);
+ }
+ if (attachInfo.mPendingAnimatingRenderNodes != null) {
+ final int count = attachInfo.mPendingAnimatingRenderNodes.size();
+ for (int i = 0; i < count; i++) {
+ registerAnimatingRenderNode(
+ attachInfo.mPendingAnimatingRenderNodes.get(i));
+ }
+ attachInfo.mPendingAnimatingRenderNodes.clear();
+ attachInfo.mPendingAnimatingRenderNodes = null;
+ }
+ final FrameInfo frameInfo = root.getUpdatedFrameInfo();
+ // Skip GPU draw; the DrawFrameTask still runs syncFrameState which pushes the
+ // staging DisplayList to active.
+ setSyncOnlyNextFrame();
+ final long syncStartNanos = profileDlReadiness ? System.nanoTime() : 0;
+ final int syncResult = syncAndDrawFrame(frameInfo);
+ root.agentOnSyncForAgentComplete(mRootNode.hasDisplayList(), syncResult);
+ if (profileDlReadiness) {
+ root.agentDlProfileOnActiveSyncComplete(
+ syncResult, System.nanoTime() - syncStartNanos);
+ }
+ }
+
void draw(View view, AttachInfo attachInfo, DrawCallbacks callbacks) {
attachInfo.mViewRootImpl.mViewFrameInfo.markDrawStart();
diff --git a/core/java/android/view/View.java b/core/java/android/view/View.java
index 1c6bd4268ca2..0ac2dec6ce92 100644
--- a/core/java/android/view/View.java
+++ b/core/java/android/view/View.java
@@ -24067,6 +24067,7 @@ public class View implements Drawable.Callback, KeyEvent.Callback,
@UnsupportedAppUsage(maxTargetSdk = Build.VERSION_CODES.R, trackingBug = 170729553)
public RenderNode updateDisplayListIfDirty() {
final RenderNode renderNode = mRenderNode;
+ boolean agentDisplayListRecorded = false;
if (!canHaveDisplayList()) {
// can't populate RenderNode, don't try
return renderNode;
@@ -24138,6 +24139,7 @@ public class View implements Drawable.Callback, KeyEvent.Callback,
} finally {
renderNode.endRecording();
setDisplayListProperties(renderNode);
+ agentDisplayListRecorded = true;
}
} else {
if ((mPrivateFlags4 & PFLAG4_HAS_VIEW_PROPERTY_INVALIDATION)
@@ -24153,6 +24155,10 @@ public class View implements Drawable.Callback, KeyEvent.Callback,
}
mPrivateFlags4 &= ~PFLAG4_HAS_MOVED;
mFrameContentVelocity = -1;
+ // The current RenderNode is closed here. An ancestor RecordingCanvas may remain on the
+ // stack; allowing a foreground frame at this boundary is the aggressive checkpoint whose
+ // reentrancy and latency are evaluated by the node-checkpoint experiment.
+ ViewRootImpl.agentNodeCheckpoint(this, agentDisplayListRecorded);
return renderNode;
}
@@ -25879,6 +25885,7 @@ public class View implements Drawable.Callback, KeyEvent.Callback,
}
notifyAppearedOrDisappearedForContentCaptureIfNeeded(true);
+ ViewRootImpl.agentNodeCheckpoint(this);
}
private boolean hasParentWantsFocus() {
@@ -28615,6 +28622,7 @@ public class View implements Drawable.Callback, KeyEvent.Callback,
mMeasureCache.put(key, ((long) mMeasuredWidth) << 32 |
(long) mMeasuredHeight & 0xffffffffL); // suppress sign extension
+ ViewRootImpl.agentNodeCheckpoint(this);
}
/**
diff --git a/core/java/android/view/ViewRootImpl.java b/core/java/android/view/ViewRootImpl.java
index 40d6318c831f..7625b7bd9c79 100644
--- a/core/java/android/view/ViewRootImpl.java
+++ b/core/java/android/view/ViewRootImpl.java
@@ -204,6 +204,7 @@ import android.os.Handler;
import android.os.IBinder;
import android.os.Looper;
import android.os.Message;
+import android.os.MessageQueue;
import android.os.ParcelFileDescriptor;
import android.os.Process;
import android.os.RemoteException;
@@ -342,6 +343,10 @@ public final class ViewRootImpl implements ViewParent,
private static final boolean DEBUG_IMF = false || LOCAL_LOGV;
private static final boolean DEBUG_CONFIGURATION = false || LOCAL_LOGV;
private static final boolean DEBUG_FPS = false;
+ // Agent on-demand observe diagnostics (settle steps, yields). Off by default;
+ // perfetto trace sections (agentObserve/agentSettlePass/agentRecordPhase2/
+ // agentYieldToForeground) carry the profiling signal regardless of this flag.
+ private static final boolean DEBUG_AGENT = false;
private static final boolean DEBUG_INPUT_STAGES = false || LOCAL_LOGV;
private static final boolean DEBUG_KEEP_SCREEN_ON = false || LOCAL_LOGV;
private static final boolean DEBUG_CONTENT_CAPTURE = false || LOCAL_LOGV;
@@ -727,6 +732,220 @@ public final class ViewRootImpl implements ViewParent,
/** Set to true while in performTraversals for detecting when die(true) is called from internal
* callbacks such as onMeasure, onPreDraw, onDraw and deferring doDie() until later. */
boolean mIsInTraversal;
+ // Agent forced-traversal no-draw mode: record DisplayList + sync staging->active,
+ // but skip GPU draw (zero GPU/SF). Set only on the UI thread for the duration of
+ // a forceTraversalForAgent() call.
+ boolean mAgentNoDraw;
+ // Agent PERSISTENT no-draw (GPU bypass): when true, EVERY performDraw() on this
+ // window skips the GPU draw -- not only the on-demand observe frame (mAgentNoDraw),
+ // but also the app's own real-VSYNC-driven traversals while the window is unfrozen
+ // and active. This is the "unfreeze period has traversal but all GPU is bypassed"
+ // mode: the view tree still measures/layouts/records its DisplayList (so on-demand
+ // DL perception stays correct and the app's state machine advances on input), but
+ // the RenderThread never rasterizes a frame for this window (zero GPU / zero SF
+ // buffer queued). Independent of freeze. Gated to agent UIs only.
+ //
+ // Tri-state per-window override, mirroring mAgentDecoupleOverride:
+ // AUTO(0) -> follow the PROCESS-WIDE default sAgentPersistentNoDrawDefault
+ // (see below): a newly-created agent window inherits whatever mode
+ // the agent last set process-wide, so sub-activities/dialogs the
+ // agent navigates to AFTER task start are covered automatically
+ // (fixes multi-window nav apps e.g. Zhihu question->answer, whose
+ // new ViewRootImpls previously rendered at full GPU).
+ // ON(1) -> force on for this window (agent UI only).
+ // OFF(-1) -> force off for this window.
+ // Actual state decided by isAgentPersistentNoDrawActive(); see performDraw().
+ // NOTE: unlike decouple, the AUTO default is OFF (process default starts false),
+ // because nodraw BLANKS the window's output -- B-GUI / screenshot configs that
+ // render the agent VD for ImageReader capture must NOT be blanked. The agent
+ // opts in by calling gpubypass, which flips the process default ON.
+ static final int AGENT_NODRAW_AUTO = 0;
+ static final int AGENT_NODRAW_ON = 1;
+ static final int AGENT_NODRAW_OFF = -1;
+ int mAgentPersistentNoDrawOverride = AGENT_NODRAW_AUTO;
+ // Process-wide default for agent-UI persistent no-draw, inherited by every new
+ // agent ViewRootImpl at construction (so newly navigated agent windows bypass
+ // GPU without per-window re-engagement). Flipped by the gpubypass/nogpubypass
+ // dumpsys backdoor. Starts false so non-bypass configs (B-GUI screenshot) are
+ // unaffected until the agent explicitly opts in.
+ static volatile boolean sAgentPersistentNoDrawDefault = false;
+ // A window must complete at least one REAL draw() to build its RenderNode
+ // DisplayList before persistent no-draw may safely skip draws -- otherwise
+ // syncForAgent() records an empty/uninitialized DL and the agent perceives a
+ // blank screen. Set true after the first real draw in performDraw(); until then
+ // isAgentPersistentNoDrawActive() returns false so the first frame renders
+ // normally (builds the DL), then subsequent frames bypass GPU. This lets newly
+ // navigated agent windows self-bootstrap: 1 real frame, then nodraw.
+ boolean mAgentHasDrawnOnce;
+ // Agent freeze: when true this window stops scheduling its own traversal/draw on
+ // real VSYNC (app self-rendering for this window halts, zero GPU). The agent still
+ // updates the DisplayList on demand via forceTraversalForAgent(), whose traversals run
+ // through the agent checkpoint wrapper. Gated to agent displays only (never display 0).
+ boolean mAgentFrozen;
+ // Agent dual-phase: skip draw-command recording in Phase 1 (measure+layout only).
+ boolean mAgentSkipRecord;
+ // Per-window invalidate counter: incremented on every invalidate() call.
+ // Used by forceTraversalForAgent to detect when the view tree has settled.
+ long mAgentDirtyTick;
+ // Per-window profiling counters for E3 experiment (reset via 'fresh reset').
+ // mAgentPhase1Count: measure+layout-only traversals during settle phase.
+ // mAgentPhase2Count: full record traversals (one per forceTraversalForAgent call).
+ // mAgentSettled: true if the last call exited via early-break, false if bound was hit.
+ int mAgentPhase1Count;
+ int mAgentPhase2Count;
+ boolean mAgentSettled;
+
+ // === Agent UI identification (launch-origin, NOT foreground/display-based) ===
+ // An "agent UI" is a UI instance launched by an agent via the agent-display API
+ // (C1), as opposed to a "main UI" launched by user interaction. Per design, the
+ // distinction is by launch origin and must follow windows derived from an agent
+ // UI (e.g. dialogs/sub-activities the agent navigates to), even if such a window
+ // were to surface on the physical display.
+ //
+ // Current identification: a window is agent UI iff it lives on a non-default
+ // display. This holds because agent UIs are launched on agent VirtualDisplays
+ // and the user's UI owns the default display; navigation-spawned windows inherit