Skip to content

feat(jira): add JIRA_SKIP_UNPARSEABLE_ISSUES to continue collectIssues on parse failure (#8949) - #9048

Closed
kotwal-itpro wants to merge 1 commit into
apache:mainfrom
kotwal-itpro:feat/jira-skip-unparseable-issues-8949
Closed

feat(jira): add JIRA_SKIP_UNPARSEABLE_ISSUES to continue collectIssues on parse failure (#8949)#9048
kotwal-itpro wants to merge 1 commit into
apache:mainfrom
kotwal-itpro:feat/jira-skip-unparseable-issues-8949

Conversation

@kotwal-itpro

Copy link
Copy Markdown

Closes #8949.

Problem

On large Jira instances, jira:collectIssues occasionally receives an otherwise-successful (2xx) response whose body cannot be decoded as JSON — typically a truncated body from a dropped connection, or (in rare misconfigured-proxy cases) an HTML body served with a 200. The current ResponseParser returns an error, which aborts the entire subtask and kills the whole Jira collection even though thousands of other pages succeeded.

This is distinct from HTTP error statuses (502/503/504), which are already handled by the collector's retry logic — the request is specifically about parse failures on successful responses.

Fix

Introduce a JIRA_SKIP_UNPARSEABLE_ISSUES env var:

  • default (unset or anything other than "true"): existing behaviour — surface the parse error and fail the subtask.
  • "true" (case-insensitive): log a warning and skip the unparseable page so the rest of the collection completes. The warning includes body length and a 256-byte body prefix so operators can still investigate without hiding the problem.

The V2 (Server) and V3 (Cloud) collectors previously duplicated the response-parsing closure. Factored the shared logic into a parseIssuesResponse(logger) helper so the new behaviour applies to both.

Test

Adds Test_parseIssuesResponse in issue_collector_test.go covering:

  • Happy path returns the expected issues, no warning.
  • Default (env var unset) surfaces the parse error, no warning.
  • JIRA_SKIP_UNPARSEABLE_ISSUES=true returns an empty non-nil slice and logs exactly one warning.
  • Case-insensitivity (TRUE).
  • Explicit "false" behaves like default.

Uses t.Setenv for isolation. Full plugins/jira/tasks suite still passes.

…s on parse failure (apache#8949)

On large Jira instances, jira:collectIssues occasionally receives an
otherwise-successful (2xx) response whose body cannot be decoded as JSON —
typically a truncated body from a dropped connection or, in rare
misconfigured-proxy cases, an HTML body served with a 200. The default
ResponseParser returns an error, which aborts the entire subtask and kills the
whole Jira collection even though thousands of other pages succeeded. This is
distinct from HTTP error statuses, which are already handled by the
collector's retry logic; the request is specifically about parse failures on
successful responses.

Introduce a JIRA_SKIP_UNPARSEABLE_ISSUES env var:
  - default (unset or anything other than "true"): existing behaviour —
    surface the parse error and fail the subtask.
  - "true" (case-insensitive): log a warning and skip the unparseable page so
    the rest of the collection completes. The warning includes body length
    and a 256-byte prefix so operators can still investigate.

Share the parser between the V2 (Server) and V3 (Cloud) collectors via a new
parseIssuesResponse(logger) helper. Tests cover the happy path, the default
error path, the skip-flag success path, case-insensitivity, and an explicit
"false" value.
@kotwal-itpro

Copy link
Copy Markdown
Author

Superseded by #9026 (which uses taskCtx.GetConfigReader().GetBool(...) instead of os.Getenv, a cleaner integration with DevLake's config plumbing) — the same feature has now landed on main. Closing this. Thanks for the reviewer bandwidth.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature][jira] Add JIRA_SKIP_UNPARSEABLE_ISSUES to continue collection when an issue page can't be parsed

1 participant