Skip to content

fix(run): serve cached sources from the resolved document, not the unwritten output path - #2128

Open
ThomasRooney wants to merge 2 commits into
mainfrom
fix/local-schema-path-download
Open

ThomasRooney wants to merge 2 commits into
mainfrom
fix/local-schema-path-download

Conversation

@ThomasRooney

@ThomasRooney ThomasRooney commented Sep 2, 2026 •

Copy link
Copy Markdown
Member

Why

Several workflows fail every speakeasy run -t all --frozen-workflow-lockfile with

failed to get schema contents: failed to download OpenAPI schema: failed to download file: <clone>/.speakeasy/temp/output_<hash>.yaml: Get "<clone>/.speakeasy/temp/output_<hash>.yaml": unsupported protocol scheme ""

They share one workflow shape: a source with an overlay (or several inputs / transforms, i.e. not IsSingleInput()) that is used by more than one target, run with --frozen-workflow-lockfile. The first target generates fine; every later target fails.

Root cause

internal/run/source.go:

  • runSourceInner skips writeToOutputLocation when w.FrozenWorkflowLock is set (if !w.FrozenWorkflowLock { ... }), on purpose: frozen runs must not touch the user's output: file. sourceRes.OutputPath is still set to Source.GetOutputLocation(), which for a non-single-input source is .speakeasy/temp/output_<sha256(inputs)[:6]>.yaml. In a frozen run that file is never written; the real document is the overlay_*.yaml / merge_*.yaml temp file that runSourceInner returns.
  • The RunSource cache added in feat: add source reference resolution and merge optimizations #1917 (if c, ok := w.SourceResults[sourceID]; ok && c.OutputPath != "" { return c.OutputPath, ... }) hands OutputPath to the second and later targets sharing the source.
  • speakeasy-core/openapi.GetSchemaContents classifies local vs remote with os.Stat(path) == nil; the missing temp file falls through to url.Parse + download.DownloadFile, hence unsupported protocol scheme "".

The same cache also served a "successful" result after a linting failure (OutputPath is set before linting), which silently disabled the quickstart minimum-viable-spec retry: retryWithMinimumViableSpec mutates the source and calls RunSource again, but got the cached un-fixed document back.

What changed

  • SourceResult records the document runSourceInner actually produced (documentPath, set only on success). The cache (cachedSource) serves that path instead of OutputPath.
  • This changes what later targets receive in every mode, not just frozen runs. Before, a cache hit returned OutputPath (the user's output: file or .speakeasy/temp/output_<hash>.yaml); now it returns the temp document the pipeline produced (overlay_*.yaml / merge_*.yaml / transform_*.yaml, or the input itself for a single-input source). In non-frozen runs the two files have identical content (writeToOutputLocation copies or reformats one into the other), so the only observable difference is the path the generator is pointed at. writeToOutputLocation itself is unchanged: non-frozen runs still write output:, frozen runs still do not.
  • The same fix covers single remote/registry input sources in frozen mode: there OutputPath is the registry_<hash> / download path that NewFrozenSource does not write either, so later targets sharing such a source hit the same missing-file error before this change.
  • Only completed runs are cached; a failed source is run again by the next caller. The in-flight entry is dropped once the run finishes so a retry is not short-circuited by a stale result, and sourceOrder no longer records a re-run source twice.
  • RunSource's slow path (runSourceOnce) re-checks the cache under sourceInflightMu before registering a run. Dropping the in-flight entry after a run had opened a check-then-delete window in which a caller that missed the cache just before the first run published could register a second run of the same source. runSourceInner publishes to SourceResults before the in-flight entry is removed, so the re-check under the registration lock closes the window; lock order is sourceInflightMu -> sourceMu only.

No speakeasy-core change is needed; the os.Stat classifier in core is correct once it is given a file that exists.

Validation

go build ./... && go vet ./internal/run/... && go test -race ./internal/run/... pass. Tests in internal/run/source_cache_test.go:

  • TestCachedSource_*: unit tests of the cache lookup (a cached result with a missing OutputPath returns the existing resolved document, which openapi.GetSchemaContents reads locally with isRemote == false; an incomplete result is not cached).
  • TestRunSource_*: drive the real RunSource pipeline over testdata/openapi.yaml + testdata/overlay.yaml with FrozenWorkflowLock set. A source shared by two targets runs once and the second target gets an existing document that differs from the never-written OutputPath; eight concurrent callers share one run; a late caller that missed the cache before the first run completed is served that run (this test fails on the previous commit, where the source ran twice); a source failing mid-pipeline is re-run and recorded once in sourceOrder; a source failing on an unknown source ref is re-run once the workflow is fixed, as the minimum-viable-spec retry does.

End-to-end check with the CLI built from this branch against a workflow of the failing shape (one source = 1 input + 5 overlays, shared by csharp, java and typescript targets, frozen lockfile):

  • Before (main): the first target generates, the second fails with failed to get schema contents: ... unsupported protocol scheme "" on the cached .speakeasy/temp/output_<hash>.yaml path.
  • After: all three targets generate (Running target ... for each), no schema-download error. The one remaining failure in that workflow is an unrelated pnpm install problem in the TypeScript target's workspace.

Summary by cubic

Fixes the source cache so targets sharing a source in --frozen-workflow-lockfile runs get the resolved document instead of the unwritten output path, which previously failed with unsupported protocol scheme "" for every target after the first.

  • Only completed runs are cached; a failed source is run again by the next caller.
  • The in-flight entry is dropped after the run finishes, which also restores the quickstart minimum-viable-spec retry.
  • The cache is re-checked under the in-flight lock before registering a run, so a late caller can't start a duplicate one.

Written for commit 50f7c2a. Summary will update on new commits.

Review in cubic

…written output path

With --frozen-workflow-lockfile the source pipeline never writes
Source.GetOutputLocation() (.speakeasy/temp/output_<hash>.yaml), yet the
RunSource cache introduced in #1917 handed that path to every target after
the first one that shares the source. The generator stats the missing temp
file, falls through to the remote download branch of GetSchemaContents and
fails with 'unsupported protocol scheme ""'.

Record the document runSourceInner actually produced on the SourceResult and
serve that from the cache. Only completed runs are cached: a failed source is
run again by the next caller, which also restores the minimum viable spec
retry (it mutates the source before re-running). The in-flight entry is
removed once a run finishes for the same reason.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 2 files

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.

Re-trigger cubic

…nSource end to end

Review follow-ups for #2128.

Dropping the in-flight entry after a run opened a check-then-delete window:
a caller that missed SourceResults just before the first run published its
result could reach the in-flight check after the entry was deleted and
register a second run of the same source. RunSource's slow path now lives in
runSourceOnce, which re-checks the cache under sourceInflightMu before
registering a run. runSourceInner publishes to SourceResults before RunSource
removes the in-flight entry, so a missing entry means either "never ran" or
"result already cached", and the re-check under the registration lock is
sufficient. Lock order is sourceInflightMu -> sourceMu only; nothing takes
them the other way round.

Add tests that drive the real RunSource pipeline over the testdata fixtures
with FrozenWorkflowLock set:
- a source with an overlay shared by two targets is run once, and the second
  target gets an existing resolved document that differs from the never
  written OutputPath;
- eight concurrent callers share one run;
- a late caller that missed the cache before the first run completed
  (runSourceOnce directly) is served the completed run; this test fails on
  the previous commit;
- a source failing mid-pipeline is re-run by the next caller and recorded
  once in sourceOrder;
- a source failing before its pipeline starts (unknown source ref) is re-run
  once the workflow is fixed, as the minimum-viable-spec retry does.
@ThomasRooney

Copy link
Copy Markdown
Member Author

Review follow-ups addressed in 50f7c2a:

Check-then-delete race on the in-flight entry (Codex P2). RunSource's slow path is now runSourceOnce, which re-checks the cache under sourceInflightMu before registering a run. runSourceInner publishes to SourceResults before RunSource removes the in-flight entry, so a missing entry means either "never ran" or "result already cached", and the re-check under the registration lock is enough to never start a second run for a completed source. Lock order is sourceInflightMu -> sourceMu only (the defer in runSourceInner and validateDocument take sourceMu alone), so there is no inversion. On the source-ref errgroup path (runSourceInner ~L185-200) a diamond such as A -> {B, C} -> D now either joins D's in-flight run or is served D's published result; a failed D is still re-run by the next caller by design (nothing is cached for it), which is what the minimum-viable-spec retry relies on.

Tests cover the fixed path (medium). source_cache_test.go now drives the real RunSource pipeline over testdata/openapi.yaml + testdata/overlay.yaml with FrozenWorkflowLock set: a source shared by two targets runs once and the second target gets an existing document that differs from the never-written OutputPath; eight concurrent callers share one run; a late caller that missed the cache before the first run completed (runSourceOnce directly) is served that run, and this test fails on 7eef08d with the source running twice; a source failing mid-pipeline (malformed overlay) is re-run by the next caller and recorded once in sourceOrder; a source failing on an unknown source ref is re-run once the workflow is fixed. The tests t.Chdir into a temp dir so .speakeasy/temp never lands in the source tree, hence nolint:paralleltest.

PR description (low). Updated: the cache now hands later targets the pipeline's temp document in every mode (identical content to output: in non-frozen runs, only the path differs); writeToOutputLocation itself is unchanged; and single remote/registry input sources in frozen mode had the same missing-file bug and are covered by the same fix.

Verification: go build ./... && go vet ./internal/run/... && go test -race ./internal/run/... all pass. golangci-lint could not be run locally (the installed binary is built with Go 1.25, module targets 1.26); CI will cover it.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant