[MODEL] support gemma4_unified#2921
Merged
Merged
Conversation
Gemma 4 unified (multimodal) checkpoints fail to quantize with "Unsupport model_type gemma4_unified". Add a dedicated definition that reuses the composite Gemma 4 decoder layout (per-projection q/k/v norms, dual pre/post feed-forward norms) but drops the per-layer input adapters this variant does not have, and refreshes the rope (cos, sin) per layer during cached replay so the sliding/full attention boundaries stay correct. Fixes ModelCloud#2920 Signed-off-by: HaozheZhang6 <zhang.hz6666@gmail.com>
Qubitium
approved these changes
Jun 15, 2026
Qubitium
left a comment
Collaborator
There was a problem hiding this comment.
@HaozheZhang6 Thank you! LGTM.
2 tasks
2 tasks
Qubitium
pushed a commit
that referenced
this pull request
Jun 19, 2026
…2925) The gemma4_unified family also has a standalone text stack (model_type "gemma4_unified_text", Gemma4UnifiedTextModel) registered in transformers, mirroring how gemma4_text parallels gemma4. #2921 added the composite gemma4_unified but not the text-only variant, so a text-only checkpoint still fails with "Unsupport model_type gemma4_unified_text". This extracts the shared decoder layout into a constant and adds Gemma4UnifiedTextQModel rooted at model.layers (model.norm / model.rotary_emb), reusing the same per-layer rope replay helper.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
gemma4_unified(multimodal Gemma 4, e.g.OpenYourMind/gemma-4-12B-it-abliterated-uncensored) fails withUnsupport model_type gemma4_unified, and failed to auto-detect module tree. It's a close sibling of the existinggemma4composite model, but its text stack drops the per-layer input adapters, so it needs its own definition rather than reusingGemma4ForConditionalGenerationGPTQ.What Changed
Gemma4UnifiedForConditionalGenerationGPTQdefinition overmodel.language_model.layers: per-projection q/k/v norms + dual pre/post feed-forward norms (same asgemma4), but noper_layer_input_gate/post_per_layer_input_norm/per_layer_projection— this variant has none. Itslanguage_modelalso has noproject_per_layer_inputs, which is why reusing thegemma4classAttributeErrors during the calibration generate hook.prepare_layer_replay_kwargsrefreshes the rope per layer: the model builds one(cos, sin)per attentionlayer_typeand hands each decoder layer the tuple for its own type, so cached-layer replay has to regenerate it for the layer'slayer_type(sliding vs full). Without it, replay reuses a stale tuple and hits a rope shape mismatch inapply_rotary_pos_emb.gemma4_unifiedinMODEL_MAPand re-exported fromdefinitions/__init__.py.Vision/audio towers stay as base (non-quantized) modules, same as
gemma4.Tests
tests/test_gemma4_unified_support.py(CPU-only): model_type → definition resolution, module tree excludes the per-layer-input paths, and the rope refresh returns the right tuple perlayer_type.$ pytest tests/test_gemma4_unified_support.py -v tests/test_gemma4_unified_support.py::test_gemma4_unified_model_type_selects_definition PASSED tests/test_gemma4_unified_support.py::test_gemma4_unified_module_tree_excludes_per_layer_input_paths PASSED tests/test_gemma4_unified_support.py::test_gemma4_unified_replay_kwargs_refresh_position_embeddings_per_layer_type PASSED 3 passed in 3.93sI also ran a full quantize → save → reload → generate on a tiny
gemma4_unified(4-bit, group_size 128) on an A100: quantization completes across both the sliding and full attention layers, the saved model reloads withTritonV2Linearpacked layers, and generation runs without error.Review Requirements
Notes
The definition follows
gemma4.py's rope-replay approach but is kept as a separate class/file since the per-layer-input machinery there doesn't apply here. I didn't have the full 12B checkpoint on hand, so the end-to-end run used a tiny syntheticgemma4_unified— happy to validate against a specific checkpoint if you'd like.