A practical guide to finding which modules of a trained LoRA cause a specific defect (blurry skin, lost identity, ignored pose) and to fixing it without retraining.
Everything here is post-hoc weight surgery: you take a .safetensors you already trained,
produce variants with some module groups zeroed or scaled, generate with each under a fixed
seed, and measure. No GPU training, minutes per iteration.
Worked examples at the end come from two real LoRAs on Qwen-Image-2.1 (a head swap and a
Train a new LoRA on top of an existing one, with the existing one frozen.
The use case: a LoRA that already works (here, a VFX-edit LoRA) and a second task to add (body swap). Continuing training on the first LoRA destroys it — in my case 500 steps at 1e-4 turned a working scene-cut edit into an out-of-domain face and killed a relight, because ~44% of the gradient came from the new task. Freezing it and learning only the delta on top cannot forget anything, and the result ships as a separate, detachable file: load the base LoRA alone for the original task, both for the new one, and dial
Adds two flags to ai-toolkit's minimax_h3_ref2va architecture:
align_video_refs — a control video becomes a true v2v guide instead of a loose
reference: the arrangement that makes LTX-style IC-LoRA training work, applied to H3.control_latent_only — the control media reaches the DiT as latents only and is never
shown to the VLM. Faster to train, much smaller embed cache, and it closes a train/inference
gap that exists in ComfyUI today.