[FeathBow]

Description

Follows #1242.

• The Qwen3.5 GDN prefill wrappers move from pegainfer-qwen35 into pegainfer_kernels::ops next to the decode wrappers, the way feat(qwen4): share the Qwen3.5 GDN decode wrappers and prove their reuse #1159 moved those: conv1d_prefill_batch_into, the seven chunkwise stages behind gated_delta_rule_prefill_chunkwise_into, and GdrChunkwiseScratch35. pegainfer-qwen35 keeps two small functions that size the scratch from its config. No kernel changes.
• pegainfer-qwen4 gets a device half behind its own qwen4 feature. weights.rs loads one layer from the shards that hold it, checks that layer's slice of the tensor contract and uploads through the shared staged loader, which gains an expert_stack record for the fused [experts, rows, cols] tensors. layer.rs runs one linear_attention layer in prefill on the hyper-connection kernels, the GDN wrappers and the MoE block. moe.rs runs the block with the router kernel, a route metadata kernel that orders the picks by expert, one cuBLAS GEMM pair per expert over its slots (gemm_rows_span_at_into_checked), the fused SiLU, a weighted combine kernel, the shared branch and the block sum.
• tools/accuracy/dump_qwen4_layer_golden.py records the reference layer on the checkpoint's own weights, once on the accelerator and once on the CPU, at 18 stage boundaries. layer::golden_gate holds the engine against it stage by stage, with the reference's own accelerator against CPU spread as the floor, and accepts a routing difference only where the reference's logits tie within two bf16 steps.
• Nothing serves yet. launch still refuses, the attention and PLE layers and the step come next, and the routed experts are the correctness first form, about a thousand launches and one host sync per layer at 2k tokens.

Test Env

Single GPU (sm_89, x86_64) with CUDA 13.1. The layer reference was recorded with transformers 5.16.1 on the same card and, for the floor comparison, on a GH200.

Verification

	main	this PR
qwen35 HF golden gate on Qwen3.5-4B, five statistics lines	recorded	identical to main, 5 of 5 lines, 2 passed
cargo test -p pegainfer-qwen35 --features qwen35 --lib recurrent::	7 passed	7 passed
cargo clippy -p pegainfer-kernels -p pegainfer-qwen35 with qwen35, all targets, deny warnings	not rerun, the move touches only these wrappers	0 findings
cargo clippy -p pegainfer-core -p pegainfer-kernels -p pegainfer-qwen4 with qwen4, all targets, deny warnings	no device half	0 findings
cargo clippy -p pegainfer-server --no-default-features --features qwen35,qwen4	0 findings	0 findings
cargo test -p pegainfer-qwen4 --features qwen4 --lib	48 passed	49 passed
layer 0 gate against the reference recorded on the same card	no device half	1 passed

Layer 0 gate, max difference against the reference per stage, with the reference's own accelerator against CPU floor:

Stage	this PR	reference floor
attn_out	7.8e-3	3.9e-3
stream_after_attn	2.0e-3	9.8e-4
moe_out	7.8e-3	7.8e-3
stream_out	3.9e-3	5.9e-3

One of the 32 tokens swaps one expert; the reference's logits for the two are one bf16 step apart, and the reference itself routes one token differently between the sm_89 and GH200 runs.

Not run: the engine side on GH200, and the GDN decode step on a carried state.

[chatgpt-codex-connector]

Codex Review Summary: 🔄 Running

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	🔄 Running since 2026-11-10T01:09:34.196280Z	6f9e3c9	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 88bd157332

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

Separate conv-state writes from prefill reads

When this layer prefills more than one token, conv1d_prefill_kernel reads conv_state for the first kernel_size - 1 positions while its t == seq_len - 1 threads overwrite the same state in the same launch (csrc/qwen35/conv1d.cu). CUDA blocks have no execution ordering, so the fresh zero history here can be replaced by tail tokens before early rows read it, producing nondeterministic or wrong qkv_conv; perform the state update in a later launch (or otherwise snapshot/synchronize it) before using this kernel for the new Qwen4 path.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Align accepted MoE geometry with the router

With a smaller Qwen4 checkpoint, this copies the config-accepted 64-expert/top-4 geometry (explicitly accepted by config::tests::smaller_widths_still_pass) into Qwen4MoeGeometry, but the first route call's validate() only accepts 512 experts/top-10. Detection and loading therefore succeed while layer prefill always errors, defeating the intended smaller-model support; either reject these dimensions in Config or make routing support them.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Preserve the single-file checkpoint fallback

For a valid unsharded checkpoint containing model.safetensors, the shared load_shard_info returns the one path with an empty weight map so find_tensor can scan that shard. This check therefore rejects the empty used list before the file is mapped, making every single-file Qwen4 checkpoint fail with "no tensor under ... in the checkpoint index"; special-case the empty map by selecting the sole shard and relying on header/contract lookup.

Useful? React with 👍 / 👎.

[FeathBow]

@codex check again please