Skip to content

Commit b79f4ce

Browse files
huacnleeclaude
andauthored
chore: Fix the CI metadata parse and skip unchanged snapshots (longbridge#2992)
Two problems with the weekly **Release GPUI** run. ## The publish step could not parse cargo metadata The run failed before it reached crates.io: SyntaxError: JSON Parse error: Unrecognized token " " at verifyKitAgainstStaging (script/bump-gpui.ts:1761:27) spawn() merged stderr into stdout and the caller parsed that merged text as the output of cargo metadata. Cargo reports its progress on stderr, and actions-rust-lang/setup-rust-toolchain sets CARGO_TERM_COLOR=always, so in CI the JSON arrived behind an ANSI escape. That escape is the character the message quotes; reproduced against the unpatched script: | Captured text | JSC message | | --- | --- | | ` Updating...` + JSON | Unexpected identifier "Updating" | | `ESC[1mESC[32m UpdatingESC[0m` + JSON | Unrecognized token "" (matches CI) | Nothing showed up locally because cargo emits no colour and no progress into a warm, non-terminal pipe. run() now returns stdout alone, and the merged stream is only used to show what a failing command said. Each stream also gets its own TextDecoder, so a multi-byte character can no longer be split across the two. ## A version was spent every Sunday The cron published a new snapshot whether or not Zed had touched GPUI. The run now reads the Zed revision the newest published snapshot was cut from out of its crates.io description (`... (gpui-pre snapshot of zed@5b055fa)`), expands that abbreviated revision through the GitHub API, fetches the commit into the shallow checkout, and diffs it against the requested revision over every published crate directory plus the Zed workspace manifest, whose versions are inlined into the staged crates. The comparison covers the whole published closure rather than only crates/gpui*, because a change in sum_tree or util reaches the snapshot just as much. A run that uploads nothing, names its own version, resumes an unfinished one, or cannot establish the comparison publishes as before; --force and a new workflow_dispatch input override it. A stale target/gpui-pre/gpui-pre.json survives in the CI cache, so it is now cleared at the start of a run and a skipped run cannot have the previous summary reported as its own. ## Verification Both paths were run against a real shallow Zed clone and the live crates.io API: # HEAD at the published revision Nothing to publish: Zed has not touched the published crates since gpui-pre 0.3.3 (zed@5b055fa); pass --force to publish anyway # HEAD at zed main 4 published files changed since gpui-pre 0.3.3, among them Cargo.toml, crates/gpui/src/platform/test/platform.rs, ... [3/7] Staging a standalone workspace --self-test passes and covers the new description parsing; typos is clean. AI-generated with Claude Code, reviewed and verified by hand. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_018QLHpFmahcCbmc26bvTJpc Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent d88fc05 commit b79f4ce

2 files changed

Lines changed: 213 additions & 16 deletions

File tree

‎.github/workflows/release-gpui.yml‎

Lines changed: 11 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -3,8 +3,11 @@ name: "Release GPUI"
33
# See script/bump-gpui.ts for the publishing and validation pipeline.
44
on:
55
schedule:
6-
# Every Sunday at 18:00 Beijing time (UTC+8).
7-
- cron: "0 10 * * 0"
6+
# Monday 00:00 US Pacific, where Zed's week starts: 08:09 UTC, a few
7+
# minutes off the hour because GitHub delays schedules that begin one,
8+
# and an hour later in Pacific daylight time. The run stops without
9+
# publishing when Zed did not change any of the republished crates.
10+
- cron: "9 8 * * 1"
811
workflow_dispatch:
912
inputs:
1013
rev:
@@ -15,6 +18,11 @@ on:
1518
description: "Publish exactly this version instead of the next <VERSION>.<N>"
1619
required: false
1720
default: ""
21+
force:
22+
description: "Publish even when Zed did not change the published crates"
23+
type: boolean
24+
required: false
25+
default: false
1826

1927
concurrency:
2028
group: ${{ github.workflow }}
@@ -52,7 +60,7 @@ jobs:
5260
env:
5361
CARGO_REGISTRY_TOKEN: ${{ secrets.CARGO_REGISTRY_TOKEN }}
5462
run: |
55-
bun script/bump-gpui.ts ${{ inputs.version }} --rev "${{ inputs.rev || 'main' }}"
63+
bun script/bump-gpui.ts ${{ inputs.version }} --rev "${{ inputs.rev || 'main' }}" ${{ inputs.force && '--force' || '' }}
5664
- name: Summary
5765
if: always()
5866
run: |

‎script/bump-gpui.ts‎

Lines changed: 202 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -35,11 +35,17 @@
3535
* Usage:
3636
* script/bump-gpui.ts [VERSION] [--rev REV] [--zed PATH]
3737
* [--dry-run] [--stage-only] [--no-verify] [--no-wait]
38+
* [--force]
3839
*
3940
* Every crate is published at `<VERSION>.<N>`, e.g. `0.3.12`: the
4041
* `VERSION` constant below (major.minor) plus a patch number that continues
4142
* from whatever crates.io already has. A positional VERSION overrides that
4243
* for one run.
44+
*
45+
* A version is only spent on a snapshot that differs from the newest
46+
* published one: the run compares the requested Zed revision against the one
47+
* that snapshot was cut from and stops when nothing under the published
48+
* crates changed. `--force` publishes regardless.
4349
*/
4450

4551
import {
@@ -82,6 +88,10 @@ const DEPENDENCY_OVERRIDES: Record<
8288
};
8389

8490
const ZED_GIT_URL = "https://github.com/zed-industries/zed";
91+
const ZED_GITHUB_API = ZED_GIT_URL.replace(
92+
"https://github.com/",
93+
"https://api.github.com/repos/",
94+
);
8595
const ZED_DEFAULT_REV = "main";
8696
const PUBLISH_PREFIX = "gpui-pre";
8797
const ROOT_CRATES = ["gpui", "gpui_platform", "gpui_macros", "reqwest_client"];
@@ -150,12 +160,25 @@ async function run(cmd: string[], options: RunOptions = {}): Promise<string> {
150160
? `(cd ${options.cwd} && ${cmd.join(" ")})`
151161
: cmd.join(" ");
152162
console.log(dim(`$ ${shown}`));
153-
const { code, output } = await spawn(cmd, options.cwd, !options.capture);
163+
const { code, stdout, output } = await spawn(
164+
cmd,
165+
options.cwd,
166+
!options.capture,
167+
);
154168
if (code !== 0) {
155169
const detail = output.trim() ? `\n${output.trim()}` : "";
156170
throw new BumpError(`command failed (${code}): ${cmd[0]}${detail}`);
157171
}
158-
return output;
172+
return stdout;
173+
}
174+
175+
/** Run a command, returning its standard output, or `undefined` if it failed. */
176+
async function runQuiet(
177+
cmd: string[],
178+
cwd: string,
179+
): Promise<string | undefined> {
180+
const { code, stdout } = await spawn(cmd, cwd, false);
181+
return code === 0 ? stdout : undefined;
159182
}
160183

161184
/** Run a command, streaming its output while also capturing it. */
@@ -167,25 +190,34 @@ async function runStreaming(
167190
return spawn(cmd, cwd, true);
168191
}
169192

193+
/**
194+
* Run a command, keeping its standard output apart from the merged stream.
195+
*
196+
* Cargo reports its progress (`Updating crates.io index`, `Compiling ...`) on
197+
* stderr, so a caller that parses `cargo metadata` has to read `stdout` on its
198+
* own; the merged `output` is there to show what a failing command said.
199+
*/
170200
async function spawn(cmd: string[], cwd: string | undefined, echo: boolean) {
171201
const process_ = Bun.spawn(cmd, {
172202
cwd,
173203
stdin: "inherit",
174204
stdout: "pipe",
175205
stderr: "pipe",
176206
});
177-
const chunks: string[] = [];
178-
const decoder = new TextDecoder();
179-
const pump = async (stream: ReadableStream<Uint8Array>) => {
207+
const out: string[] = [];
208+
const merged: string[] = [];
209+
const pump = async (stream: ReadableStream<Uint8Array>, sink?: string[]) => {
210+
const decoder = new TextDecoder();
180211
for await (const chunk of stream) {
181212
const text = decoder.decode(chunk, { stream: true });
182-
chunks.push(text);
213+
sink?.push(text);
214+
merged.push(text);
183215
if (echo) process.stdout.write(text);
184216
}
185217
};
186-
await Promise.all([pump(process_.stdout), pump(process_.stderr)]);
218+
await Promise.all([pump(process_.stdout, out), pump(process_.stderr)]);
187219
const code = await process_.exited;
188-
return { code, output: chunks.join("") };
220+
return { code, stdout: out.join(""), output: merged.join("") };
189221
}
190222

191223
// ---------------------------------------------------------------------------
@@ -237,6 +269,122 @@ async function zedRevision(zed: string): Promise<string> {
237269
}
238270
}
239271

272+
/** The Zed revision a published description names, if it names one. */
273+
function snapshotRev(description: string): string | undefined {
274+
return /\bzed@([0-9a-f]{7,40})\b/.exec(description)?.[1];
275+
}
276+
277+
/**
278+
* Resolve a Zed revision to a commit the checkout holds.
279+
*
280+
* The checkout is fetched one commit deep, so the previously published
281+
* revision is normally absent and has to be fetched on its own. A server only
282+
* answers for a full revision and crates.io records an abbreviated one, so
283+
* GitHub expands it first. Returns `undefined` when the revision cannot be
284+
* reached, which leaves the caller with nothing to compare against.
285+
*/
286+
async function fetchZedRev(
287+
zed: string,
288+
rev: string,
289+
): Promise<string | undefined> {
290+
const local = await runQuiet(
291+
["git", "rev-parse", "--verify", "--quiet", `${rev}^{commit}`],
292+
zed,
293+
);
294+
if (local !== undefined) return local.trim();
295+
let full: string;
296+
try {
297+
const response = await fetch(`${ZED_GITHUB_API}/commits/${rev}`, {
298+
headers: {
299+
"User-Agent": USER_AGENT,
300+
Accept: "application/vnd.github+json",
301+
...(process.env.GITHUB_TOKEN === undefined
302+
? {}
303+
: { Authorization: `Bearer ${process.env.GITHUB_TOKEN}` }),
304+
},
305+
});
306+
if (!response.ok) return undefined;
307+
full = String(((await response.json()) as Toml).sha);
308+
} catch {
309+
return undefined;
310+
}
311+
const fetched = await runQuiet(
312+
["git", "fetch", "--depth", "1", "--no-tags", "origin", full],
313+
zed,
314+
);
315+
return fetched === undefined ? undefined : full;
316+
}
317+
318+
/**
319+
* The newest published snapshot, and the Zed revision it was cut from.
320+
*
321+
* Every crate records that revision in `[package.metadata.gpui-pre]`, which
322+
* crates.io does not serve, and in its description, which it does:
323+
* `... (gpui-pre snapshot of zed@5b055fa)`.
324+
*/
325+
async function publishedSnapshot(
326+
base: string,
327+
crate: string,
328+
): Promise<{ version: string; rev: string } | undefined> {
329+
const pattern = new RegExp(`^${base.replaceAll(".", "\\.")}\\.(\\d+)$`);
330+
const data = await cratesIoGet(`${crate}/versions`);
331+
let newest: { version: string; patch: number } | undefined;
332+
for (const version of data?.versions ?? []) {
333+
const match = pattern.exec(String(version.num));
334+
if (match === null || version.yanked) continue;
335+
const patch = Number(match[1]);
336+
if (newest === undefined || patch > newest.patch)
337+
newest = { version: String(version.num), patch };
338+
}
339+
if (newest === undefined) return undefined;
340+
const detail = await cratesIoGet(`${crate}/${newest.version}`);
341+
const rev = snapshotRev(String(detail?.version?.description ?? ""));
342+
return rev === undefined ? undefined : { version: newest.version, rev };
343+
}
344+
345+
/**
346+
* The newest published snapshot when Zed has not touched it since, so that a
347+
* run publishes a version only when there is something in it.
348+
*
349+
* The comparison covers the directory of every crate that gets published,
350+
* together with Zed's workspace manifest, whose `[workspace.dependencies]`
351+
* versions are inlined into the staged crates. Whatever it cannot establish —
352+
* nothing published yet, a revision GitHub will not expand, a checkout without
353+
* the Zed remote — publishes.
354+
*/
355+
async function unchangedSnapshot(
356+
zed: string,
357+
crates: Crate[],
358+
zedSha: string,
359+
): Promise<{ version: string; rev: string } | undefined> {
360+
const gpui = crates.find((crate) => crate.name === "gpui");
361+
if (gpui === undefined) return undefined;
362+
const published = await publishedSnapshot(VERSION, gpui.publishedName);
363+
if (published === undefined) return undefined;
364+
const previous = await fetchZedRev(zed, published.rev);
365+
if (previous === undefined) {
366+
logWarn(
367+
`Cannot reach zed@${published.rev}, the revision ${PUBLISH_PREFIX} ` +
368+
`${published.version} was cut from; publishing without comparing`,
369+
);
370+
return undefined;
371+
}
372+
if (previous === zedSha) return published;
373+
const paths = ["Cargo.toml", ...crates.map((crate) => crate.relDir)];
374+
const changed = await runQuiet(
375+
["git", "diff", "--name-only", previous, zedSha, "--", ...paths],
376+
zed,
377+
);
378+
if (changed === undefined) return undefined;
379+
const files = changed.split("\n").filter((line) => line !== "");
380+
if (files.length === 0) return published;
381+
logInfo(
382+
`${files.length} published files changed since ${PUBLISH_PREFIX} ` +
383+
`${published.version}, among them ${files.slice(0, 3).join(", ")}`,
384+
);
385+
return undefined;
386+
}
387+
240388
// ---------------------------------------------------------------------------
241389
// Workspace model
242390
// ---------------------------------------------------------------------------
@@ -1292,6 +1440,13 @@ fn helper(input: TokenStream) -> TokenStream { input }
12921440
if (!pathRewriter.includes(expected))
12931441
throw new BumpError(`self-test path rewriter is missing \`${expected}\``);
12941442
}
1443+
for (const [description, expected] of [
1444+
["Zed's GPU-accelerated UI framework (gpui-pre snapshot of zed@5b055fa)", "5b055fa"],
1445+
["no revision here", undefined],
1446+
] as const) {
1447+
if (snapshotRev(description) !== expected)
1448+
throw new BumpError(`self-test read the wrong revision from \`${description}\``);
1449+
}
12951450
logSuccess("Facade-aware gpui_macros transformation self-test passed");
12961451
}
12971452

@@ -1501,7 +1656,10 @@ async function cratesIoGet(path: string): Promise<Toml | undefined> {
15011656
* that number, and the run resumes at it instead of starting a new one.
15021657
* Yanked versions still occupy their number.
15031658
*/
1504-
async function nextVersion(base: string, crates: Crate[]): Promise<string> {
1659+
async function nextVersion(
1660+
base: string,
1661+
crates: Crate[],
1662+
): Promise<{ version: string; resuming: boolean }> {
15051663
const pattern = new RegExp(`^${base.replaceAll(".", "\\.")}\\.(\\d+)$`);
15061664
const numbers = new Map<Crate, Set<number>>();
15071665
let highest = -1;
@@ -1516,17 +1674,17 @@ async function nextVersion(base: string, crates: Crate[]): Promise<string> {
15161674
highest = Math.max(highest, ...published);
15171675
await Bun.sleep(200); // be polite to the crates.io API
15181676
}
1519-
if (highest < 0) return `${base}.0`;
1677+
if (highest < 0) return { version: `${base}.0`, resuming: false };
15201678
const incomplete = crates.filter(
15211679
(crate) => !numbers.get(crate)!.has(highest),
15221680
);
15231681
if (incomplete.length > 0) {
15241682
logInfo(
15251683
`Resuming ${base}.${highest}: ${incomplete.length} crates are still missing it`,
15261684
);
1527-
return `${base}.${highest}`;
1685+
return { version: `${base}.${highest}`, resuming: true };
15281686
}
1529-
return `${base}.${highest + 1}`;
1687+
return { version: `${base}.${highest + 1}`, resuming: false };
15301688
}
15311689

15321690
async function versionIsPublished(
@@ -1657,6 +1815,7 @@ Options:
16571815
--no-verify skip \`cargo publish --dry-run\` verification
16581816
--no-wait abort instead of waiting on the crates.io rate limit
16591817
--skip-kit-check do not build and test this repository against the staged crates
1818+
--force publish even when Zed did not change the published crates
16601819
-h, --help show this help
16611820
`;
16621821

@@ -1669,6 +1828,7 @@ interface Args {
16691828
noVerify: boolean;
16701829
noWait: boolean;
16711830
skipKitCheck: boolean;
1831+
force: boolean;
16721832
selfTest: boolean;
16731833
}
16741834

@@ -1686,6 +1846,7 @@ function parseCommandLine(argv: string[]): Args {
16861846
"no-verify": { type: "boolean", default: false },
16871847
"no-wait": { type: "boolean", default: false },
16881848
"skip-kit-check": { type: "boolean", default: false },
1849+
force: { type: "boolean", default: false },
16891850
"self-test": { type: "boolean", default: false },
16901851
help: { type: "boolean", short: "h", default: false },
16911852
},
@@ -1713,6 +1874,7 @@ function parseCommandLine(argv: string[]): Args {
17131874
noVerify: parsed.values["no-verify"] as boolean,
17141875
noWait: parsed.values["no-wait"] as boolean,
17151876
skipKitCheck: parsed.values["skip-kit-check"] as boolean,
1877+
force: parsed.values.force as boolean,
17161878
selfTest: parsed.values["self-test"] as boolean,
17171879
};
17181880
}
@@ -1804,6 +1966,7 @@ async function main(argv: string[]): Promise<number> {
18041966
const totalSteps = args.stageOnly ? 4 : args.dryRun ? 6 : 7;
18051967
logHeader(`Publishing GPUI from Zed as ${PUBLISH_PREFIX}`);
18061968
mkdirSync(WORK_DIR, { recursive: true });
1969+
rmSync(join(WORK_DIR, "gpui-pre.json"), { force: true });
18071970

18081971
logStep(`1/${totalSteps}`, "Preparing the Zed checkout");
18091972
const zed = await prepareZed(args.rev, args.zed);
@@ -1814,13 +1977,39 @@ async function main(argv: string[]): Promise<number> {
18141977
logStep(`2/${totalSteps}`, "Collecting the crates that gpui needs");
18151978
const ws = loadWorkspace(zed);
18161979
const crates = selectCrates(ws);
1817-
const version = args.version ?? (await nextVersion(VERSION, crates));
1980+
const next =
1981+
args.version === undefined
1982+
? await nextVersion(VERSION, crates)
1983+
: { version: args.version, resuming: false };
1984+
const version = next.version;
18181985
const width = Math.max(...crates.map((c) => c.name.length));
18191986
for (const crate of crates)
18201987
console.log(` ${crate.name.padEnd(width)} -> ${crate.publishedName}`);
18211988
logSuccess(
18221989
`${crates.length} crates will be published as version ${bold(version)}`,
18231990
);
1991+
1992+
// A run that uploads nothing, that was told which version to publish, or
1993+
// that resumes an unfinished one always does its work; the weekly cron is
1994+
// the one that must not spend a version on a snapshot nobody changed.
1995+
if (
1996+
!args.force &&
1997+
!args.dryRun &&
1998+
!args.stageOnly &&
1999+
args.version === undefined &&
2000+
!next.resuming
2001+
) {
2002+
const unchanged = await unchangedSnapshot(zed, crates, zedSha);
2003+
if (unchanged !== undefined) {
2004+
console.log();
2005+
logSuccess(
2006+
"Nothing to publish: Zed has not touched the published crates since " +
2007+
`${PUBLISH_PREFIX} ${unchanged.version} (zed@${unchanged.rev}); ` +
2008+
"pass --force to publish anyway",
2009+
);
2010+
return 0;
2011+
}
2012+
}
18242013
console.log();
18252014

18262015
logStep(`3/${totalSteps}`, "Staging a standalone workspace");

0 commit comments

Comments
 (0)