fix: multimodal-LM (Qwen3.5_5) support for ex-LRP#1
Open
mann1x wants to merge 13 commits into
Open
Conversation
…, add LRP mask test, and clean up files
…t multiple architectures, and enforce explicit lrp_scores
…back, add lru_cache for memory safety, and clean up unused LRP parameters
…restore model state after LRP backward pass
…lxt to pyproject extras
…olate LRP score output directories, and patch notebook paths
…ft-padded attention masks, and fix pipeline device arguments
…process interpreter - Fix CUDA device fallback: Normalize device selection in LRPConfig.__post_init__ to gracefully fall back to CPU when CUDA is unavailable - Fix Qwen architecture dispatch: Use exact model_type matching instead of substring to avoid false positives (qwen3, qwen2_moe) - Fix subprocess interpreter: Use sys.executable instead of bare 'python' to ensure correct venv/conda environment
…ation, float16 underflow - Fix weight normalization: Remove total_weight division to respect user-supplied weights and match documented merge formula (Masked task vector × weight) - Fix cached LRP scores: Return deep copy from _load_lrp_scores to prevent cross-task tensor mutation and aliasing - Fix float16 underflow: Always use float32 for model loading to prevent gradient underflow in backward pass (float16 min normal ~6e-5 causes small gradients to zero) - Add warning message about float32 requirement for accurate LRP computation
The total_weight computation and zero-check were dead code after removing weight normalization. This misleading validation could confuse future maintainers into 'fixing' the apparent bug by re-introducing normalization, silently changing merge semantics. Removed: - total_weight = sum(self.model_weights.values()) - Zero-check validation The merge now clearly uses un-normalized per-model weights as documented.
…script location - Fix silent warnings: validate() now raises FileNotFoundError immediately if model paths don't exist, preventing confusing subprocess errors later - Fix script path resolution: compute_lrp_scores() now resolves lrp_computer.py relative to the pipeline script's directory using __file__, allowing the pipeline to be run from any working directory - Add clear error messages with actionable guidance for both issues
End-to-end fixes that make the ex-LRP method in this PR work against
multimodal models like Qwen3_5ForConditionalGeneration. Validated on
Qwen3.5-4B (jackrong-v2 + crow-4b → base) producing a clean Q6_K with
HumanEval pass@1 = 51.22% and MBPP pass@1 = 49.40%.
Five minimal patches:
* mergekit/architecture/base.py
Pydantic v2 forward-references in ConfiguredModuleArchitecture and
ConfiguredModelArchitecture aren't resolved eagerly; add
model_rebuild() at module load. Without this, the first model load
fails with PydanticUserError: not fully defined.
* mergekit/architecture/auto.py
Make 'optional' layer-aware. Hybrid-attention archs (Qwen3.5_5
alternates full / linear attention) have tensors like dt_bias only
in some layers. The original _wi flagged optional based on a single
layer's presence; widen to true if missing in *any* layer.
* mergekit/merge_methods/lrp.py
Replace strict raise with base-passthrough when LRP scores are
missing for a tensor. Multimodal LRP is computed only on the
language_model branch; vision tower / MTP heads have no relevance
signal and should retain base weights instead of failing the merge.
* mergekit/config.py
Allow `str` in the ParameterSetting union so per-source params like
`lrp_scores: \"/path/to/scores.safetensors\"` parse cleanly via
mergekit-yaml.
* lrp_computer.py
- Recognize qwen3_5_text inner-LM model_type (the inner LM of
Qwen3_5ForConditionalGeneration) and dispatch AttnLRP against it.
- Clone tied tensors (lm_head.weight ↔ model.embed_tokens.weight)
by data_ptr() before save_file, since safetensors save rejects
shared-storage tensors.
Reproducer + LRP signal artefacts published at
https://huggingface.co/ManniX-ITA/Qwen3.5-4B-M4-ex-LRP
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
4 tasks
Owner
|
@copilot resolve the merge conflicts in this pull request |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
End-to-end fixes that make this PR's ex-LRP method work against multimodal models like
Qwen3_5ForConditionalGeneration. Five minimal patches acrossmergekit/architecture/{base,auto}.py,mergekit/merge_methods/lrp.py,mergekit/config.py, andlrp_computer.py. Each patch includes a one-paragraph rationale in the commit body.Without these patches, ex-LRP cannot be invoked on any multimodal LM (vision tower + language model + MTP heads); it fails at first model-load (pydantic), or at graph-execution (missing tensors in hybrid-attention layers), or at save (tied tensors), or in the LRP score computation (model_type not recognized).
What's tested
Validated end-to-end on Qwen3.5-4B with two source fine-tunes merged into base via
mergekit-yamldriving thelrpmethod (this PR's recipe), with LRP relevance scores precomputed via thelrp_computer.py(also patched). All scoring done vialm_evallocal-chat-completions/local-completionsagainstllama-serverrunning the Q6_K quantization, temperature 0,max_gen_toks=2048.The M5 row is the apples-to-apples LRP comparison vs M4 — same LRP scores, different merger. M4 (ex-LRP via this PR) and M5 (OMv2 + LRP via a custom merger) bracket the merger effect; they share an identical importance signal.
Reproducer artefacts (HF)
All weights, GGUFs, READMEs, importance signals (Fisher .safetensors, LRP .safetensors), and the merge YAMLs / configs used in the test runs are published:
lrp/lrp_config.yamlis the exact mergekit-yaml input.Each repo's README has the full 5-way comparison table for cross-reference.
Companion lxt patch
The
lrp_computer.pychange in this PR depends onlxt(LRP-eXplains-Transformers) being importable in environments withoutvit_torch. A separate one-line PR has been opened at rachtibat/LRP-eXplains-Transformers#42 wrapping that import in try/except.Test plan
convert_hf_to_gguf.py+llama-quantizeNote
Patches kept as small and orthogonal as possible; happy to split into separate commits if preferred. The
lrp_computer.pychange is the largest by line count because two related issues collapse into one file (model_type dispatch + tied-tensor save).