Dpr pr2b - #2
Draft
randyspauldingamd wants to merge 25 commits into
Draft
Conversation
…ification Prerequisite base (Layer 0 / S) for enabling Dapper in TheRock. Makes the dependency mapping work from a single-gtest build (no discrete binaries) and adds an extensible attribution-bridge interface plus a runner fallback classification. - enhanced_ninja_parser: additively emit synthetic bin/test_<stem> keys for test/gtest source objects so file->fixture selection works whether the build is single-gtest (one exe) or discrete (idempotent there). Record compiled_sources. Factor _project_relative/_attribute_object_deps; split build_mapping/export_mapping so bridges can run over the in-memory maps before export. - main.py: add `parse --bridges=<list>`; lazy-import registry (stem->common_stem, symbol->symbol_graph), precedence (symbol supersedes stem), per-bridge timing. Bridge modules live on the gap-fix branches; base branch runs with no bridge. - selective_test_filter: emit fallback_mode (union|entire_category|minimal) so an unattributed-but-compiled change runs the entire category filter (never the minimal default) while dapper stays subtractive-only. - dapper.cmake: MIOPEN_DAPPER_BRIDGES cache var threaded into the parse target(s). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… suffices dapper_init() force-set MIOPEN_TEST_DISCRETE=1 because the original dependency parser needed the ~280 discrete test binaries to build the file->test mapping. With single-gtest support (per-source synthetic bin/test_<stem> keys in enhanced_ninja_parser), the mapping is derived from the single aggregated miopen_gtest, so the discrete build is no longer required. Drop the forced set so the build respects the user/CI MIOPEN_TEST_DISCRETE setting (default off) -- avoiding the expensive discrete build by default. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…l-iteration) Adds a local-iteration mode (cache var MIOPEN_DAPPER_USE_CACHED, default ON) that reuses already-present dapper input files and skips both regeneration and the test-binary build: - dapper.cmake: pass --use-cached to shas/fixtures/parse when the var is ON, and drop dapper_mapping's miopen-tests dependency in that case (no build). When OFF, the full generation runs and depends on miopen-tests. - main.py (shas, parse) and extract_gtest_fixtures.py: with --use-cached, reuse the existing output if present (skip), else fail fast with a clear message telling the user to reconfigure once with -DMIOPEN_DAPPER_USE_CACHED=OFF to populate the inputs. Intended for fast local re-runs / iterating on the analysis without rebuilding. TheRock must always regenerate on the builder; the var is forced OFF there (handled in the TheRock guard, Layer 2). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
If a gtest shard process exits before writing its XML (e.g. it crashes), that shard's output is absent. Previously _convert_xml_shards aborted on the first missing shard (not listing the rest) and analyze_sharded_gtest silently skipped missing shards and produced a partial analysis -- which is misleading. Now dapper collects ALL shards that produced no output, reports exactly which ones failed (name + expected path), and aborts with a non-zero exit before doing any analysis. No partial results. Shared abort_missing_shards() is used by both the xml->json conversion path and the analysis reader. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ests too)
miopen_gtest_sharded_dapper depended on the parent test/ directory tests + shards,
but not on tests registered in test/gtest/ itself -- notably
${TEST_NAME}_hip_graph_serial -- so that GPU serial test could run after the dapper
analysis and bury its summary. Also depend on the current directory's TESTS
property (shards + hip_graph_serial) so the analysis is truly last.
TheRock-only category suites are registered later by apply_test_category_labels and
do not run in MICI, so they are intentionally not required here.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Additive `symbol` bridge (src/symbol_graph.py, selected via -DMIOPEN_DAPPER_BRIDGES=symbol; supersedes stem). Runs after the ninja-deps mapping. Uses nm to build a provider->consumer symbol graph: each source is attributed to the test sources whose objects reference an out-of-line symbol that source defines -- mirroring what the linker pulls in. This is the semantically exact attribution for compiled .cpp bodies (a body can only affect another TU via a symbol it references) and, unlike the stem heuristic, also handles library .cpp bodies and symbols declared in non-sibling headers, with no over/under- attribution. Additive layer over the include graph (which still covers header-only/template/ macro code with no out-of-line symbol); never removes edges. Only project- compiled objects are scanned; out-of-source deps are skipped. Robust to the base parser's normalizer name (_to_project_relative or _project_relative). Validated: attribution logic (incl. library .cpp) via stubbed nm, and _nm_symbols parsing against real GNU nm (undefined/global captured, locals excluded). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…2 part 1) Runner-side, transparent-to-CTest activation of Dapper (the reduced subtractive set actually runs). Pure Python; no CMake yet. - shared/ctest/dapper_union.py: self-contained union computation honoring fallback_mode (union -> impact filter intersected with the category; entire_category -> the whole category filter; minimal -> smoke default). Subtractive-only. No dependency-parser imports so it can ship next to the binary. - projects/miopen/scripts/run_miopen_gtest.py: thin gtest_runner wrapper (the path the yaml already names) that computes the filter and execs the gtest binary with --gtest_filter, propagating the exit code. Imports the co-installed dapper_union. - shared/ctest/parse_test_categories.py: activate the dormant gtest_runner hook. New --dapper-json enables it; categories with a truthy enable_dapper (and the yaml's gtest_runner) emit an install add_test that calls the wrapper instead of the binary directly -- for both the main and per-arch GPU-exclusion suites. Baseline (no --dapper-json) output is unchanged. Tested: all three fallback modes, exit-code propagation, and install-line generation with/without --dapper-json (incl. GPU-exclusion suites). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…t 2) Wires Dapper into the CMake/CTest flow so it is active in TheRock and switchable in the native build. - MIOPEN_DAPPER_MODE cache var (off|validate|union; TheRock default union, native default validate) + MIOPEN_DAPPER_BASE_REF. 'off' fully disables Dapper. - test/gtest/CMakeLists.txt: in TheRock, when mode!=off, generate the impact JSON on the GPU-less builder and install it; in union mode pass --dapper-json to the category generator and co-install run_miopen_gtest.py + dapper_union.py next to the CTestTestfile. Native build gates dapper_init on mode!=off. - dapper.cmake: dapper_therock_generate_json() -- build-time shas/fixtures/parse (--bridges) /select producing miopen_dapper_tests.json (dapper_filter + fallback_mode), using source-tree script paths and --source-dir so git runs in the MIOpen source worktree (out-of-source build). - shared/ctest/TestCategories.cmake: DAPPER_JSON arg -> --dapper-json. - main.py / selective_test_filter.py: --base-ref and --source-dir for shas/select; git runs via -C <source>; undetermined diff (missing base ref / shallow history) fails open to fallback_mode=entire_category instead of crashing. - dapper_union.py: fail open to the entire category if the JSON is missing at run time. - test_categories.yaml: opt 'standard' into Dapper (enable_dapper: True). Tested (Python): source-dir shas/select against real git, fail-open path, and install-line generation from the real yaml (standard + GPU-exclusion suites route through the wrapper; quick stays direct). CMake wiring is not yet exercised by a real build. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…igures cleanly The shard-file writes, dapper_dev_filters(), and dapper_add_sharded_test() were gated only on NOT MIOPEN_BUILD_IN_THEROCK, so -DMIOPEN_DAPPER_MODE=off in a native build would reference dapper_init()-defined vars/macros that were never created and fail at configure. Introduce MIOPEN_ENABLE_DAPPER_NATIVE (ON only for a native build with Dapper enabled) and gate all three sites on it; default MIOPEN_TEST_SINGLE_GTEST when dapper_init() is skipped. Native 'validate' (default) and TheRock paths are unchanged; 'off' now cleanly disables Dapper while the single gtest and its shard tests still build and run. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…er/runner split Reference for the Dapper selective-gtest flow: modes (off/validate/union), core concepts (dapper_filter/union_filter/fallback_mode), the TheRock builder->runner cycle (impact analysis on the GPU-less builder; union + execution on the GPU runner), the native MIOpen-CI validate cycle, the stem/symbol attribution bridges, configuration knobs, file map, and known limitations. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
MIOPEN_DAPPER_USE_CACHED is a local-iteration tool that reuses existing dapper inputs. The TheRock builder must always regenerate the impact JSON from scratch, so force the cache var OFF in the TheRock guard branch (it can never leak in via -D), and dapper_therock_generate_json never passes --use-cached. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…mework The symbol (nm-based) bridge is correctness-dominant and is the only attribution bridge we ship. Remove the 'stem' heuristic bridge references from the registry, cmake help, and docs while keeping the generic BRIDGE_REGISTRY/BRIDGE_SUPERSEDES/--bridges machinery for future bridges. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Make MIOPEN_DAPPER_BRIDGES default to 'symbol' (nm-based, correctness- dominant) so the attribution gap for common-.cpp-body changes is closed out of the box for both native/MICI and TheRock. Set to empty to disable. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ysis HipGraphExist is filtered out of the parallel shards (LLVM JIT OOM under concurrent GPU pressure) and re-registered as a standalone RUN_SERIAL test. Its XML was not in SHARDS_FILE, so dapper_diff never ingested it: a serial-test failure still failed ctest but was invisible to the dapper compliance report. Append the serial XML to SHARDS_FILE (native dapper + GPU only). select records the path into the tests JSON; dapper_diff then requires it and aborts if absent. Ordering is already guaranteed by dapper_add_sharded_test(), which orders the analysis dead-last after every current-dir test, the serial test included. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…for production) MIOPEN_DAPPER_USE_CACHED was a local-iteration convenience to reuse existing dapper input files and skip regeneration/build. It has no place in production CI (TheRock already forced it OFF), so remove it entirely: - dapper.cmake: drop the cache var, the _dapper_cached_flag, and the cached-mode dependency branch; mapping always depends on dapper_shas + miopen-tests. - test/gtest/CMakeLists.txt: drop the TheRock forced-OFF override. - main.py / extract_gtest_fixtures.py: drop the --use-cached args and the skip-if-present / error-if-missing logic; generators always regenerate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Roll the Dapper mode switch, single-gtest defaulting, and native/TheRock dispatch out of test/gtest/CMakeLists.txt and into a single dapper_init() orchestrator in dapper.cmake. CMakeLists.txt now keeps only the TheRock-detection block, the MIOPEN_ENABLE_DAPPER_NATIVE gate, and an include + dapper_init() call. - dapper.cmake: new dapper_init() (mode/base-ref/bridges cache vars + dispatch); the former dapper_init() body is renamed _dapper_native_init(); the single-gtest default helper moves here as _dapper_default_single_gtest(). - Fix: MIOPEN_DAPPER_BRIDGES is now set on the shared path, so the 'symbol' default actually applies to TheRock too (previously only the native init set it, leaving TheRock's parse running with --bridges= empty). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…cate copies) dapper_union.py duplicated the pattern-split/overlap helpers and the union-core loop from miopen_gtest_runner.py. There is no reason to keep separate copies for MICI and TheRock: the math is identical. - Move dapper_union.py into the dependency-parser package (src/) as the canonical, stdlib-only union module (splitting/overlap + subtractive intersection + fallback). - miopen_gtest_runner.calc_union_filter now imports compute_union_filter and delegates the math (dropping its duplicated split_gtest_filter_includes/_fixed_prefix/ patterns_overlap and the inline loop; also drops dead matches_any_filter). It keeps only the native shard-JSON/XML I/O. Union output is unchanged. - TheRock installs the same file (from dependency-parser/src) next to the wrapper; it stays import-free of the package so it stands alone on the runner. - run_miopen_gtest.py local-testing fallback now points at dependency-parser/src. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
dapper_json_file was a leftover placeholder from an earlier TheRock implementation that read the impact-JSON path from the yaml. The current design passes it through CMake (DAPPER_JSON -> --dapper-json), so the yaml key is read nowhere. Remove it. The gtest_runner and enable_dapper keys are retained -- they are actively consumed by parse_test_categories.py. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ild) 'resource' is a Unix-only stdlib module, so a top-level 'import resource' breaks the TheRock Windows build with ModuleNotFoundError. extract_gtest_fixtures.py (run on the builder) imported it but never used it -- drop the dead import. all_gtest_ fixtures.py actually uses it to disable core dumps; guard the import (try/except) and make disable_core_dump() a no-op where 'resource' is absent. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… work Dapper's build-time tooling (Python + nm / C preprocessor) is not Windows-ready. Force MIOPEN_DAPPER_MODE=off on Windows hosts in dapper_init(); the existing 'off' path then skips all dapper wiring so the build falls back to the normal full-category test flow. Document the concerns to resolve when porting: nm vs llvm-nm in the symbol bridge, shlex/preprocessor handling in extract_gtest_fixtures, and other Unix-only assumptions. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…retain originals Restore develop's behavior for TheRock: ctest invokes the binary directly with the filter burned into the add_test command -- no runtime wrapper, no dapper code shipped to the runner. Only the filter *value* changes for dapper categories in union mode. - main.py: new 'finalize-ctest' subcommand (build time, builder). For each Dapper-enabled category it computes the union (reusing dapper_union.resolve_filter, honoring fallback_mode), burns it into '<name>_suite', adds '<name>_original_suite' retaining the full filter, and records category_<NAME>_filter + category_<NAME>_union in the JSON (downloadable record). Atomic writes; fails open (full category) if inputs unreadable. - dapper_union.py: split resolve_filter (pure) out of compute_filter for reuse. - dapper.cmake: dapper_therock_generate_json() now runs shas/fixtures/parse/select then finalize-ctest in one build-time command, and installs the finalized CTestTestfile + reference JSON. Called from CMakeLists after apply_test_category_labels (so the install CTestTestfile exists). dapper_init no longer calls it. - test/gtest/CMakeLists.txt: minimal -- one macro call + a union-mode install guard. - Revert shared/ctest (parse_test_categories.py, TestCategories.cmake) to develop: dapper adds nothing there now. Remove the inert gtest_runner key from test_categories.yaml. - Delete scripts/run_miopen_gtest.py (+ scripts/); dapper_union.py no longer ships to the runner. Rewrite DAPPER.md for the burn-in flow. All dapper file production is build-time, single-process, atomic; nothing dapper runs or writes at ctest time (safe under the 4x parallel ctest / shared-FS MICI model). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The build-time finalize adds a suite that keeps the full (pre-union) category filter alongside the dapper-reduced '<name>_suite'. Rename it from '<name>_original_suite' to '<name>_unfiltered_suite' (clearer: it is the unfiltered/full run). Updates the finalize-ctest logic, help/docstrings, cmake comments, and DAPPER.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
github-actions
Bot
added
documentation
dapper_diff's compliance verdict (COMPLIANT/FAIL/NOT VIABLE) runs only in native validate mode, which has the full shard run to compare against. TheRock union mode just runs the burned-in reduced set with nothing verifying coverage. Record this gap in DAPPER.md Known limitations as future work. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
just previewing changes without impacting ROCm#9617