Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions .github/workflows/quality-gate-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -273,7 +273,7 @@ jobs:
echo "::error::Expected go compiler to report undefined: undefinedVar"
exit 1
}
[[ "$ERROR_MESSAGE" == *"#### Error"* && "$ERROR_MESSAGE" == *"What was expected:"* && "$ERROR_MESSAGE" == *"How to fix it:"* ]] || {
[[ "$ERROR_MESSAGE" == *"Error:"* && "$ERROR_MESSAGE" != *"What was expected:"* ]] || {
echo "::error::Expected the Go typecheck diagnostic template"
exit 1
}
Expand Down Expand Up @@ -441,7 +441,7 @@ jobs:
echo "::error::Expected basedpyright to identify __init__.py with reportReturnType error"
exit 1
}
[[ "$ERROR_MESSAGE" == *"#### Error"* && "$ERROR_MESSAGE" == *"What was expected:"* && "$ERROR_MESSAGE" == *"How to fix it:"* ]] || {
[[ "$ERROR_MESSAGE" == *"Error:"* && "$ERROR_MESSAGE" != *"What was expected:"* ]] || {
echo "::error::Expected the typecheck diagnostic template"
exit 1
}
Expand Down Expand Up @@ -722,7 +722,7 @@ jobs:
echo "::error::Expected TypeScript compiler to report error TS2322 (type mismatch)"
exit 1
}
[[ "$ERROR_MESSAGE" == *"#### Error"* && "$ERROR_MESSAGE" == *"How to fix it:"* ]] || {
[[ "$ERROR_MESSAGE" == *"Error:"* && "$ERROR_MESSAGE" != *"What was expected:"* && "$ERROR_MESSAGE" != *"See the typecheck diagnostics"* ]] || {
echo "::error::Expected the TypeScript typecheck diagnostic template"
exit 1
}
Expand Down
15 changes: 4 additions & 11 deletions quality-gate/docs/output-summary-layouts.md
Original file line number Diff line number Diff line change
Expand Up @@ -232,33 +232,26 @@ project configuration, and TypeScript framework.
### Quality Gate: Type Check (<language/framework>)
Outcome: failure

#### Error

Error:
```text
<diagnostic output provided by the failed tool>
```

What was expected:
```text
<the failed command should exit successfully>
```

How to fix it:
```text
<tool-provided hint, or a neutral command to rerun locally>
<tool-provided remediation, when available>
```
````

Notes for discussion:

- The template is static, but error and remediation contents come from the tool whenever available.
- The expected section states only the command success contract; it must not invent a root cause.
- The remediation section is shown only when the tool provides a hint or suggested command.
- Tool hints such as `uv`'s `hint:` lines are preserved in the remediation section.
- One box represents one failed command or phase. Multiple diagnostics from that command remain together.
- One error section represents one failed command or phase. Multiple diagnostics from that command remain together.
- For typecheck summaries, setup, download, and generic recent-output noise must not be copied into the summary.
- Diagnostic output is bounded; the complete tool output remains in the job logs.
- If a tool provides no remediation hint, show the exact command and neutral guidance to resolve the reported diagnostics.
- If a tool provides no remediation hint, only the error section is shown.

## Test

Expand Down
14 changes: 3 additions & 11 deletions quality-gate/docs/testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,8 +56,8 @@ action fails and that the reported message is exact. This keeps configuration
errors actionable instead of reducing them to an unexplained exit code.

Typecheck failures use a compact diagnostic template. Tool output and hints are
copied dynamically, while the action supplies only the section labels and the
command-success expectation. Setup output, download progress, and the generic
copied dynamically, while the action supplies only the section labels. Setup
output, download progress, and the generic
recent-output fallback are not included in typecheck summaries. Other checks
retain their existing recent-output fallback for compatibility with their
diagnostic assertions.
Expand Down Expand Up @@ -153,22 +153,14 @@ the existing detection error is included after `Error:`.

Outcome: failure

#### Error

Error:
```text
<diagnostic output provided by basedpyright>
```

What was expected:
```text
<basedpyright command> must exit successfully (exit 0).
```

How to fix it:
```text
Run the command locally and resolve the diagnostics shown above:
<basedpyright command>
<tool-provided remediation, when available>
```

## Test
Expand Down
4 changes: 1 addition & 3 deletions quality-gate/go/scripts/check.sh
Original file line number Diff line number Diff line change
Expand Up @@ -186,16 +186,14 @@ case "$CHECK" in
run_strlint
;;
app:typecheck)
qg_run_typecheck go 'go vet ./...' go vet ./...
qg_run_typecheck go go vet ./...
if [[ -d cmd ]]; then
qg_run_typecheck \
go \
'go build -ldflags="-s -w" -o /dev/null ./cmd/...' \
go build -ldflags='-s -w' -o /dev/null ./cmd/...
else
qg_run_typecheck \
go \
'go build -ldflags="-s -w" -o /dev/null ./...' \
go build -ldflags='-s -w' -o /dev/null ./...
fi
;;
Expand Down
8 changes: 1 addition & 7 deletions quality-gate/python/scripts/check.sh
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,6 @@ sync_project() {
if [[ "$CHECK" == app:typecheck ]]; then
qg_run_typecheck \
uv \
'uv sync --frozen --all-groups' \
uv sync --frozen --all-groups
else
uv sync --frozen --all-groups
Expand Down Expand Up @@ -241,10 +240,7 @@ case "$CHECK" in
if [[ ! -f pyproject.toml ]]; then
qg_typecheck_error \
quality-gate \
'app:typecheck' \
'app:typecheck requires a pyproject.toml with uv configuration. setup.py-only projects are not supported for this check.' \
'A Python typecheck requires pyproject.toml with uv configuration.' \
'Add pyproject.toml with uv configuration, then rerun app:typecheck.'
'app:typecheck requires a pyproject.toml with uv configuration. setup.py-only projects are not supported for this check.'
exit 1
fi
sync_project
Expand All @@ -253,12 +249,10 @@ case "$CHECK" in
}; then
qg_run_typecheck \
basedpyright \
"uvx basedpyright@${BASEDPYRIGHT_VERSION} --project $PROJECT_DIR" \
uvx "basedpyright@${BASEDPYRIGHT_VERSION}" --project "$PROJECT_DIR"
else
qg_run_typecheck \
basedpyright \
"uvx basedpyright@${BASEDPYRIGHT_VERSION} --project $ACTION_DIR/pyrightconfig.json ." \
uvx "basedpyright@${BASEDPYRIGHT_VERSION}" \
--project "$ACTION_DIR/pyrightconfig.json" .
fi
Expand Down
46 changes: 18 additions & 28 deletions quality-gate/scripts/lib/diagnostics.sh
Original file line number Diff line number Diff line change
Expand Up @@ -12,16 +12,21 @@ qg_error() {
qg_typecheck_output() {
local tool=$1
local output=$2
local diagnostic

case "$tool" in
uv)
local diagnostic
diagnostic=$(printf '%s\n' "$output" | awk '
/^error:/ { started=1 }
/^hint:/ { exit }
started { print }
')
diagnostic=$(printf '%s\n' "$diagnostic" | sed -E 's/[[:space:]]+To (create|update|install|run) .*([Rr]un `.*)$//')
[[ -n "$diagnostic" ]] && printf '%s\n' "$diagnostic" || printf '%s\n' "$output"
;;
project-info)
printf '%s\n' "$output" | sed -E 's/\.[[:space:]]+(Add|Commit|Update|Run) .*/./'
;;
*)
printf '%s\n' "$output"
;;
Expand All @@ -44,6 +49,8 @@ qg_typecheck_hint() {

if [[ "$tool" == uv ]]; then
printf '%s\n' "$output" | sed -nE 's/^.*([Rr]un `.*)$/\1/p'
elif [[ "$tool" == project-info ]]; then
printf '%s\n' "$output" | sed -nE 's/^.*\. (Add|Commit|Update|Run) (.*)$/\1 \2/p'
fi
}

Expand All @@ -58,53 +65,36 @@ qg_typecheck_limit() {

qg_typecheck_error() {
local tool=$1
local command=$2
local output=$3
local expected=${4:-"$command must exit successfully (exit 0)."}
local fix=${5:-}
local output=$2
local diagnostic hint message

diagnostic=$(qg_typecheck_output "$tool" "$output")
hint=$(qg_typecheck_hint "$tool" "$output")
diagnostic=$(qg_typecheck_limit "$diagnostic")
hint=$(qg_typecheck_hint "$tool" "$diagnostic")
if [[ -z "$fix" ]]; then
if [[ -n "$hint" ]]; then
fix="$hint"
else
fix=$(cat <<EOF
Run the command locally and resolve the diagnostics shown above:
$command
EOF
)
fi
fi

message=$(cat <<EOF
#### Error

Error:
\`\`\`text
$diagnostic
\`\`\`

What was expected:
\`\`\`text
$expected
\`\`\`
EOF
)
if [[ -n "$hint" ]]; then
message+=$(cat <<EOF

How to fix it:
\`\`\`text
$fix
$hint
\`\`\`
EOF
)
fi
qg_error "$message"
}

qg_run_typecheck() {
local tool=$1
local command=$2
shift 2
shift
local output status

if output=$("$@" 2>&1); then
Expand All @@ -114,6 +104,6 @@ qg_run_typecheck() {
status=$?
fi

qg_typecheck_error "$tool" "$command" "$output"
qg_typecheck_error "$tool" "$output"
return "$status"
}
11 changes: 2 additions & 9 deletions quality-gate/scripts/render-summary.sh
Original file line number Diff line number Diff line change
Expand Up @@ -121,21 +121,14 @@ render_detected_languages() {
}

render_typecheck_error() {
if grep -q '^#### Error ' "$ERROR_FILE"; then
if grep -q '^Error:' "$ERROR_FILE"; then
cat "$ERROR_FILE"
return
fi

printf '#### Error\n\n'
printf 'Error:\n```text\n'
cat "$ERROR_FILE"
printf '\n```\n\n'
printf 'What was expected:\n```text\n'
printf 'The typecheck command must exit successfully (exit 0).\n'
printf '```\n\n'
printf 'How to fix it:\n```text\n'
printf 'See the typecheck diagnostics in the job logs and resolve the reported errors.\n'
printf '```\n'
printf '\n```\n'
}

printf '%s\n' "$SUMMARY_NAME" > "$NAME_FILE"
Expand Down
42 changes: 29 additions & 13 deletions quality-gate/tests/scripts/test-portable.sh
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,8 @@ assert_contains() {
}

unset GITHUB_ACTION_PATH GITHUB_ENV GITHUB_OUTPUT GITHUB_PATH GITHUB_STEP_SUMMARY RUNNER_TEMP RUNNER_TOOL_CACHE
# shellcheck disable=SC1091
source "$ROOT/scripts/lib/diagnostics.sh"

assert_equal \
"name=go" \
Expand Down Expand Up @@ -149,15 +151,10 @@ bash "$ROOT/scripts/render-summary.sh" \
--summary-file "$SUMMARY_FILE"
assert_equal "Format (Typescript - Next.js)" "$(<"$SUMMARY_NAME_FILE")" "Next.js summary name"

(
export QUALITY_GATE_ERROR_FILE="$ERROR_FILE"
# shellcheck disable=SC1091
source "$ROOT/scripts/lib/diagnostics.sh"
qg_typecheck_error \
uv \
'uv sync --frozen --all-groups' \
$'setup noise\nerror: The lockfile needs to be updated.\nhint: To update the lockfile, run uv lock.'
) 2>/dev/null
QUALITY_GATE_ERROR_FILE="$ERROR_FILE" qg_typecheck_error \
uv \
$'setup noise\nerror: The lockfile needs to be updated.\nhint: To update the lockfile, run uv lock.' \
2>/dev/null
rm -f "$SUCCESS_FILE"
bash "$ROOT/scripts/render-summary.sh" \
--check app:typecheck \
Expand All @@ -167,14 +164,33 @@ bash "$ROOT/scripts/render-summary.sh" \
--name-file "$SUMMARY_NAME_FILE" \
--summary-file "$SUMMARY_FILE"
typecheck_summary=$(<"$SUMMARY_FILE")
assert_contains "$typecheck_summary" "#### Error" "typecheck diagnostic box"
assert_contains "$typecheck_summary" "Error:" "typecheck error section"
assert_contains "$typecheck_summary" "error: The lockfile needs to be updated." "typecheck tool diagnostic"
assert_contains "$typecheck_summary" "hint: To update the lockfile, run uv lock." "typecheck tool hint"
assert_contains "$typecheck_summary" "uv sync --frozen --all-groups" "typecheck expected command"
if [[ "$typecheck_summary" == *"Recent output:"* || "$typecheck_summary" == *"setup noise"* ]]; then
assert_contains "$typecheck_summary" "How to fix it:" "typecheck remediation section"
if [[ "$typecheck_summary" == *"What was expected:"* || "$typecheck_summary" == *"#### Error"* || "$typecheck_summary" == *"setup noise"* ]]; then
fail "typecheck summary should omit capture noise"
fi

QUALITY_GATE_ERROR_FILE="$ERROR_FILE" qg_typecheck_error \
project-info \
'Missing .node-version for TypeScript app:typecheck check. Add the Node.js major version used by the project.' \
2>/dev/null
bash "$ROOT/scripts/render-summary.sh" \
--check app:typecheck \
--language typescript \
--error-file "$ERROR_FILE" \
--success-file "$SUCCESS_FILE" \
--name-file "$SUMMARY_NAME_FILE" \
--summary-file "$SUMMARY_FILE"
project_info_summary=$(<"$SUMMARY_FILE")
assert_contains "$project_info_summary" "Missing .node-version for TypeScript app:typecheck check." "project-info diagnostic"
assert_contains "$project_info_summary" "How to fix it:" "project-info remediation section"
assert_contains "$project_info_summary" "Add the Node.js major version used by the project." "project-info remediation"
if [[ "$project_info_summary" == *"What was expected:"* || "$project_info_summary" == *"See the typecheck diagnostics"* ]]; then
fail "project-info summary should not include generated guidance"
fi

capture_env="$TEMP_DIR/bash-env"
capture_log="$TEMP_DIR/capture.log"
printf 'QUALITY_GATE_ERROR_FILE=%q\nQUALITY_GATE_OUTPUT_LOG=%q\nQUALITY_GATE_CHECK=app:lint\nsource %q\n' \
Expand Down Expand Up @@ -207,7 +223,7 @@ set -e
assert_equal "1" "$capture_status" "typecheck captured command status"
typecheck_fallback=$(<"$ERROR_FILE")
assert_contains "$typecheck_fallback" "Command failed (exit 1): false" "typecheck concise fallback"
if [[ "$typecheck_fallback" == *"Recent output:"* || "$typecheck_fallback" == *"typecheck noise"* ]]; then
if [[ "$typecheck_fallback" == *"Recent output:"* || "$typecheck_fallback" == *"typecheck noise"* || "$typecheck_fallback" == *"How to fix it:"* ]]; then
fail "typecheck fallback should omit recent output"
fi

Expand Down
Loading
Loading