Make developer-specific test-harness paths configurable via environment variables

[kaijunli-infr]

Problem

Several test harnesses hard-code paths that only exist on the original author's machine, so on a fresh clone they cannot run and their unit tests fail before exercising any logic.

Scratch root /data/ycfeng/tmp, used for large intermediate outputs by:

• tests/performance/sim_walltime_scaling/sweep.py (DEFAULT_TEMP_ROOT, which also bounds FRONTIER_WALLTIME_TMPDIR)
• tests/e2e/moe_ep_baseline_replay.py (TMP_ROOT, used for path validation, child TMPDIR, and CLI defaults)
• tests/e2e/moe_ep_non_dummy_matrix.py (default --output-root for both matrix kinds)

Pinned Reference checkout /data/ycfeng/stepfun-performance-optimization/Frontier/worktrees/ref-afd-readonly, used by the PD-AF parity harness (tests/e2e/pd_af_parity/harness.py, reference_observer_bootstrap.py) and tests/integration/test_pdaf_reference_lifecycle_observer.py.

Change

Two commits, one per path family. Both keep the historical path as the fallback when the variable is unset, so existing deployments are unchanged.

FRONTIER_TMP_ROOT (commit 1)

• Add tests/scratch_root.py with resolve_scratch_root(). The value must be absolute and is re-read on every call.
• The three harness modules resolve their root through it. FRONTIER_WALLTIME_TMPDIR keeps its semantics and must still resolve to a descendant of the (now relocatable) root. Rejection messages point at FRONTIER_TMP_ROOT as the remedy.
• Unit tests for the two affected harnesses point the root at pytest's tmp_path via an autouse fixture. New tests/unit/test_scratch_root.py covers the helper.

FRONTIER_PDAF_REFERENCE_REPO_ROOT (commit 2)

• Add tests/e2e/pd_af_parity/reference_repo_root.py with resolve_reference_repo_root().
• harness.py, reference_observer_bootstrap.py, and the integration test resolve REFERENCE_REPO_ROOT through it once at import, so the many call sites that treat it as a stable pin keep working. Only the location is configurable; the pinned git HEAD and source SHA-256 checks are unchanged. The bootstrap's mismatch error names the variable.
• test_pdaf_parity_harness_wave1.py builds its producer payload from the resolved root instead of a literal. New tests/unit/test_pdaf_reference_repo_root.py covers the helper.

AGENTS.md documents both variables under Tests and gets a Modification History row.

Verification

Fresh clone on a machine without /data/ycfeng (environment.yml + pip install -e ".[test]", plus torch/matplotlib), running tests/unit tests/integration:

Scope	Before	After
test_sim_walltime_scaling_sweep.py + test_moe_ep_baseline_replay.py	13 failed, 96 passed	0 failed, 110 passed
Overall	87 failed, 6 errors	74 failed, 6 errors

With FRONTIER_PDAF_REFERENCE_REPO_ROOT pointed at an existing plain directory, 17 of the 51 bootstrap unit tests additionally pass; the remaining 34 (and the 5 integration errors) need a real checkout at the pinned commit dcb1cc8e, which is not in this repository's public history, so they cannot pass anywhere without that private worktree. That is inherent to the identity checks and unchanged by this PR.

Other remaining failures pre-exist on main and are unrelated: docs-contract tests referencing a missing tests/debug fixture directory, and a few tests needing the profiling env.

Also verified manually: with FRONTIER_TMP_ROOT set, sweep._resolve_temp_root() returns the override and rejects FRONTIER_WALLTIME_TMPDIR=/tmp; with both variables unset, both helpers return their historical paths.

Out of scope

The /local/ycfeng/... interpreter defaults in the GPU profiling shell scripts under tests/analysis, tests/e2e/operator_parity, and tests/performance are not touched here.

[fwyc0573]

Hi Kevin, thanks for your PR. Here are some review comments I put together with the help of AI—feel free to take a look. If you have any follow‑up updates, that would be even better. Appreciate it!

Comment 1

Severity: BLOCK
File: tests/e2e/pd_af_parity/reference_observer_bootstrap.py:18-21

This import breaks the PD-AF reference integration path.

The integration test sets the child process PYTHONPATH to the pinned Reference checkout (tests/integration/ test_pdaf_reference_lifecycle_observer.py:889 and :1050). That checkout does not contain tests.e2e.pd_af_parity.reference_repo_root, so the bootstrap
exits before argument parsing with:

ModuleNotFoundError: No module named 'tests.e2e.pd_af_parity'

Please make the bootstrap self-contained under a Reference-only PYTHONPATH. Move the environment-variable parsing into the bootstrap or load a helper by
absolute file path without importing the current repository's tests package. Do not add the current repository to PYTHONPATH, because its frontier
package could override the pinned Reference runtime.

Please add a subprocess regression test that runs the observer-enabled Reference path with the same PYTHONPATH setup used by the integration test.

Comment 2

Severity: MEDIUM
File: tests/e2e/pd_af_parity/reference_repo_root.py:27-35

The Reference root is represented inconsistently across processes.

This resolver returns the raw environment-variable path. The bootstrap and integration driver call Path.resolve() before writing the path to the
sidecar, while the harness compares the sidecar value with the unresolved REFERENCE_REPO_ROOT string.

A valid path containing .. or a symlink can therefore produce a false parity mismatch. For example, the sidecar may contain /workspace/reference,
while the harness expects /workspace/link/../reference.

Please use one canonical representation everywhere. Either canonicalize the path in the shared resolver or call .resolve(strict=True) before every
producer and validator comparison.

Please add tests for:

• an override containing ..;
• an override using a symlink;
• equality between the sidecar value and the harness validation value.

Comment 3

Severity: MEDIUM
File: tests/e2e/run_moe_ep_non_dummy_matrix.sh:15-19

This wrapper bypasses the new FRONTIER_TMP_ROOT setting.

The Python entry point uses resolve_scratch_root() only when --output-root is absent. This wrapper always passes --output-root, with /data/ycfeng/ tmp/frontier_non_dummy_matrix as its default. Setting only FRONTIER_TMP_ROOT therefore has no effect when this wrapper is used.

Please pass --output-root only when MATRIX_OUTPUT_ROOT is explicitly set. Otherwise, let the Python entry point resolve the default from
FRONTIER_TMP_ROOT.

Please verify both cases:

• with FRONTIER_TMP_ROOT set and MATRIX_OUTPUT_ROOT unset, output goes to <FRONTIER_TMP_ROOT>/frontier_non_dummy_matrix;
• with MATRIX_OUTPUT_ROOT set, the explicit output directory is preserved.

Merge Requirements

1. Fix the bootstrap import failure under the Reference-only PYTHONPATH.
2. Add a subprocess regression test for the observer-enabled Reference run.
3. Make Reference root canonicalization consistent across the resolver, bootstrap, integration test, sidecar, and harness.
4. Add .. and symlink path tests.
5. Update the MoE matrix shell wrapper so FRONTIER_TMP_ROOT controls its default output path.
6. Run the focused unit tests and the full PD-AF lifecycle integration test in a real git checkout.
7. Run:

bash -n tests/e2e/run_moe_ep_non_dummy_matrix.sh
git diff --check origin/main...origin/pr-29
pytest -q tests/integration/test_pdaf_reference_lifecycle_observer.py