Carcosa: Add ECC modeling/Balar Bridge/State Tracking - #2704
Conversation
|
Status Flag 'Pre-Test Inspection' - Auto Inspected - Inspection is Not Necessary for this Pull Request. |
|
Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects: Pull Request Auto Testing STARTING (click to expand)Build InformationTest Name: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements
Build InformationTest Name: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements_Make-Dist
Build InformationTest Name: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements_MT-2
Build InformationTest Name: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements_MR-2
Build InformationTest Name: SST__Autotest_OSX-15-XC16_OMPI-4.1.6_PY3.10_sst-elements
Using Repos:
Pull Request Author: nab880 |
|
Status Flag 'Pull Request AutoTester' - Jenkins Testing: all Jobs PASSED Pull Request Auto Testing has PASSED (click to expand)Build InformationTest Name: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements
Build InformationTest Name: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements_Make-Dist
Build InformationTest Name: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements_MT-2
Build InformationTest Name: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements_MR-2
Build InformationTest Name: SST__Autotest_OSX-15-XC16_OMPI-4.1.6_PY3.10_sst-elements
|
|
Status Flag 'Pre-Merge Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging |
|
All Jobs Finished; status = PASSED, However Inspection must be performed before merge can occur... |
1 similar comment
|
All Jobs Finished; status = PASSED, However Inspection must be performed before merge can occur... |
Clarify ownership, configuration, and shared state APIs.
|
Status Flag 'Pre-Test Inspection' - Auto Inspected - Inspection is Not Necessary for this Pull Request. |
|
Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects: Pull Request Auto Testing STARTING (click to expand)Build InformationTest Name: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements
Build InformationTest Name: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements_Make-Dist
Build InformationTest Name: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements_MT-2
Build InformationTest Name: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements_MR-2
Build InformationTest Name: SST__Autotest_OSX-15-XC16_OMPI-4.1.6_PY3.10_sst-elements
Using Repos:
Pull Request Author: nab880 |
|
Status Flag 'Pull Request AutoTester' - Error: Jenkins Jobs - Timed out waiting for build 2824 to complete: Total Wait = 5460; Job Timeout = 5400 |
|
Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED Note: Testing will normally be attempted again in approx. 8 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run. Pull Request Auto Testing has FAILED (click to expand)Job: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements
Job: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements_Make-Dist
Job: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements_MT-2
Job: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements_MR-2
Job: SST__Autotest_OSX-15-XC16_OMPI-4.1.6_PY3.10_sst-elements
|
|
Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects: Pull Request Auto Testing STARTING (click to expand)Build InformationTest Name: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements
Build InformationTest Name: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements_Make-Dist
Build InformationTest Name: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements_MT-2
Build InformationTest Name: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements_MR-2
Build InformationTest Name: SST__Autotest_OSX-15-XC16_OMPI-4.1.6_PY3.10_sst-elements
Using Repos:
Pull Request Author: nab880 |
|
Status Flag 'Pull Request AutoTester' - Jenkins Testing: all Jobs PASSED Pull Request Auto Testing has PASSED (click to expand)Build InformationTest Name: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements
Build InformationTest Name: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements_Make-Dist
Build InformationTest Name: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements_MT-2
Build InformationTest Name: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements_MR-2
Build InformationTest Name: SST__Autotest_OSX-15-XC16_OMPI-4.1.6_PY3.10_sst-elements
|
|
Status Flag 'Pre-Merge Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging |
|
All Jobs Finished; status = PASSED, However Inspection must be performed before merge can occur... |
5 similar comments
|
All Jobs Finished; status = PASSED, However Inspection must be performed before merge can occur... |
|
All Jobs Finished; status = PASSED, However Inspection must be performed before merge can occur... |
|
All Jobs Finished; status = PASSED, However Inspection must be performed before merge can occur... |
|
All Jobs Finished; status = PASSED, However Inspection must be performed before merge can occur... |
|
All Jobs Finished; status = PASSED, However Inspection must be performed before merge can occur... |
|
All Jobs Finished; status = PASSED, However PR is now STALE, and must be retested. Set the AT: RETEST Label to force retest.... |
2 similar comments
|
All Jobs Finished; status = PASSED, However PR is now STALE, and must be retested. Set the AT: RETEST Label to force retest.... |
|
All Jobs Finished; status = PASSED, However PR is now STALE, and must be retested. Set the AT: RETEST Label to force retest.... |
|
All Jobs Finished; status = PASSED, However PR is now STALE, and must be retested. Set the AT: RETEST Label to force retest.... |
1 similar comment
|
All Jobs Finished; status = PASSED, However PR is now STALE, and must be retested. Set the AT: RETEST Label to force retest.... |
berquist
added
AT: RETEST
|
Status Flag 'Pull Request AutoTester' - User Requested Retest - Label AT: RETEST will be reset after testing. |
|
Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects: Pull Request Auto Testing STARTING (click to expand)Build InformationTest Name: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements
Build InformationTest Name: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements_Make-Dist
Build InformationTest Name: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements_MT-2
Build InformationTest Name: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements_MR-2
Build InformationTest Name: SST__Autotest_OSX-15-XC16_OMPI-4.1.6_PY3.10_sst-elements
Using Repos:
Pull Request Author: nab880 |
|
Status Flag 'Pull Request AutoTester' - Jenkins Testing: all Jobs PASSED Pull Request Auto Testing has PASSED (click to expand)Build InformationTest Name: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements
Build InformationTest Name: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements_Make-Dist
Build InformationTest Name: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements_MT-2
Build InformationTest Name: SST__Autotest_sst-test_OMPI-4.1.4_PY3.9_sst-elements_MR-2
Build InformationTest Name: SST__Autotest_OSX-15-XC16_OMPI-4.1.6_PY3.10_sst-elements
|
sst-autotester
removed
the
AT: RETEST
|
All Jobs Finished; status = PASSED, However Inspection must be performed before merge can occur... |
3 similar comments
|
All Jobs Finished; status = PASSED, However Inspection must be performed before merge can occur... |
|
All Jobs Finished; status = PASSED, However Inspection must be performed before merge can occur... |
|
All Jobs Finished; status = PASSED, However Inspection must be performed before merge can occur... |
|
All Jobs Finished; status = PASSED, However PR is now STALE, and must be retested. Set the AT: RETEST Label to force retest.... |
13 similar comments
|
All Jobs Finished; status = PASSED, However PR is now STALE, and must be retested. Set the AT: RETEST Label to force retest.... |
|
All Jobs Finished; status = PASSED, However PR is now STALE, and must be retested. Set the AT: RETEST Label to force retest.... |
|
All Jobs Finished; status = PASSED, However PR is now STALE, and must be retested. Set the AT: RETEST Label to force retest.... |
|
All Jobs Finished; status = PASSED, However PR is now STALE, and must be retested. Set the AT: RETEST Label to force retest.... |
|
All Jobs Finished; status = PASSED, However PR is now STALE, and must be retested. Set the AT: RETEST Label to force retest.... |
|
All Jobs Finished; status = PASSED, However PR is now STALE, and must be retested. Set the AT: RETEST Label to force retest.... |
|
All Jobs Finished; status = PASSED, However PR is now STALE, and must be retested. Set the AT: RETEST Label to force retest.... |
|
All Jobs Finished; status = PASSED, However PR is now STALE, and must be retested. Set the AT: RETEST Label to force retest.... |
|
All Jobs Finished; status = PASSED, However PR is now STALE, and must be retested. Set the AT: RETEST Label to force retest.... |
|
All Jobs Finished; status = PASSED, However PR is now STALE, and must be retested. Set the AT: RETEST Label to force retest.... |
|
All Jobs Finished; status = PASSED, However PR is now STALE, and must be retested. Set the AT: RETEST Label to force retest.... |
|
All Jobs Finished; status = PASSED, However PR is now STALE, and must be retested. Set the AT: RETEST Label to force retest.... |
|
All Jobs Finished; status = PASSED, However PR is now STALE, and must be retested. Set the AT: RETEST Label to force retest.... |
3 participants
This pull request is focused on adding Error Correction Code (ECC) fault-injection and resilience scoring framework for a GPU-accelerated Vision Language Action (VLA) inference pipeline. The VLA inference pipeline is stored in a seperate repository, but these are the needed components in Carcosa.
Main Components:
Hali — interface layer on the memory path, shares a ring with other Hali components. Intercepts memory accesses to small
control regionsand dispatches to interception agents. Works with MemEvents or MMIO StandardMem.InterceptionAgentAPI — Base for agents that intercept memory accesses.
BalarRingBridge — subcomponent in Hali, passes CUDA calls to balar/GPGPU-Sim. Stages H2D weigh payload into a dedicated region. Folds the result into a 64-bit checksum and replies Done.
EccGuard — Sits on a memory link per access it draws fault based on a fault model (poisson, jedec_mix, ect). Classigies a line as clean/correctable/DUE/escape based off ECC scheme none/secded/chipkill. Models the latency or aborts the frame.
PipelineStateRegistry/PortModuleStateGate — Provide functionality to monitor where/when faults are injected and how they affect behaviors.
How a program runs:
CPU issues MMIO read, Hali intercepts and the agent dispatches to the next kernel. It maintains the currentKernel/piplineCycle in a registry. During the GPU phase the bridge or the model GPU run the GEMM. EccGuard injects faults and classifies each access. The result is snapshotted and a checksum is published.