Skip to content

App staging re-fetches identical content on every host and every run #272

Description

@dtwersky

What

When deploying apps, the controller-side staging area is keyed per host and deleted at the end of every run. As a result, identical app sources are fetched over and over:

  • Per host: staging is keyed by inventory_hostname, so every host in a play fetches the same git repos independently. A play against N indexers clones the same repo N times.
  • Per run: the staging directory is wiped on completion (success or failure), so the next run re-fetches everything from scratch even when nothing in the source has changed.

There is no reuse of previously fetched content across hosts or across runs.

Why it matters

  • Wall-time and I/O. For large sources, the redundant clones/downloads dominate real-play time. The cost scales with host count and run frequency for zero added value.
  • Blocks safe parallelism. Because the staging path is shared per host and rebuilt each run, two concurrent plays touching the same host collide in that directory (the known clone-corruption / cleanup-race issue). Without content-addressed reuse there's no clean path to running plays in parallel.

Desired outcome

Identical apps (same git remote, repo, and version) should be fetched once and reused across hosts and across runs, instead of re-fetched every time. Staging should consume already-fetched content rather than re-downloading it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions