Reddit rules are impossible to please these days, so for image comparisons go to the Civitai article, this here will be just a dumb wall of text: https://civitai.com/articles/8733
--------
--------
About two years ago, u/ruSauron transformed my approach to LoRA training when he posted this on Reddit
Performing block weight analysis can significantly impact how your LoRA functions. For me, the main advantages include preserving composition, character pose, background, colors, overall model quality, sharpness, and reducing "bleeding."
There are some disadvantages, too: resemblance can be lost if you don’t choose the right blocks or fail to perform a thorough weight analysis. Your LoRA might need a higher strength setting to function correctly. This can be fixed with a rescale for SD1.5 and SDXL after all is done. Previously, I used AIToolkit (before it became a trainer) for this.
For some reason, just now with Flux, people have been giving blocks the proper attention. Since r/Yacben published this article, many have been using target block training for LoRA training. I’m not saying this isn’t effective—I recently tried it myself with great results.
However, if my experiments with SD1.5 hold up, training with specific blocks versus conducting a full block analysis and fine-tuning the weights after full-block training yield very different results.
IMO, adjusting blocks post-training is easier and yields better outcomes, allowing you to test each block and its combinations easily. maDcaDDie recently published a helpful workflow for this here, and I strongly recommend trying Nihedon’s fork of the LoRA Block Weight tool here.
Conducting a post block weight analysis is time-consuming, but imagine doing it for every training session with each block combination. With Flux’s 57 blocks, this would require a lifetime of training.
When I tried this with SD1.5, training directly on specific blocks didn’t produce the same results as chopping out the blocks afterward; it was as if the model “compensated,” learning everything in those blocks, including undesirable details. This undermined the advantages of block-tuning—preserving composition, character pose, background, colors, sharpness, etc. Of course, this could differ with Flux, but I doubt it.
Until recently, there was no way to save a LoRA after analyzing block weights. With ComfyUi I managed to change the weights and merge it to the model, but I could not save the lora itself. So I created my own tool here:
Some people have asked about what I consider an ideal LoRA training workflow including block weight analysis. Here are my thoughts:
- Dataset
- Captions or No Captions: This makes a big difference in SD15 and SDXL, though I’m unsure about Flux. I’ve always used captions, but recently I tried a “no-caption” person LoRA, and it worked great. The captioned version was also good, so I’m uncertain. For simple concepts or character LoRAs, captions might be needed, especially with Flux. However, for hard complex concepts with many variables and details, captions may be beneficial. I recommend using JoyCaption and Taggui for this.
- Regularizations: Another tricky topic. I found success with regularization for my “sweaty shirt” LoRA, using AIToolkit to set reg weight to only 25%. Kohya doesn’t offer this option, so I eventually gave up on regularization there; I feel like it drags the training down. Regularization might be more useful for fine-tuning. LoRAs will always bleed and they are plug and play, so no one should be worrying to much about it. Recently, my answer is “no” to regularization, but I’m not completely sure.
- Large/Small Dataset: I prefer larger datasets of 100-300 images, especially with weight blocks available. I don’t think small datasets allow for full perfect resemblance. While 10-25 images might work OK for most people, it depends on how strict you are about resemblance.
- Training Parameters
- I won’t go into detail, as parameter choice requires endless testing, and I haven’t found ideal settings for Flux yet.
- Consider longer sessions. While 1000 steps is commonly suggested, it’s often not enough for block weight analysis. I recommend at least 6000 steps, even for Flux. Of course, this depends on factors like LR, image count, regularization, etc… endless testing.
- Epoch/Strength Selection
- Please for heaven’s sake, don’t just take the last epoch, and DON’T trust your training samples for epoch choosing.
- If overcooking does occurs in later epochs, don’t discard them outright; instead, test them at a lower weight, like 0.67. Many of my best LoRAs come from later epochs with lower weights.
- Test with different prompts and scenarios.
- Use XY plots, XY plots, XY plots…. (e.g., epoch vs. weight vs. prompts)
- If you have preliminary block weight insights, you can test them here, though it’s often more practical to do this after you have the best epoch and best strength.
- Block Weight Analysis
- Use ComfyUI or the LoRA Block Weight extension. There are no hard and fast tips for this—sometimes a single block is crucial, sometimes it’s just the ALLOUT, sometimes a combination, and sometimes a block gains relevance only in conjunction with another. Trial and error is key.
- Experiment with different weights, not just 0 and 1.
- You could try XY plot here. I often prefer guessing, removing and adjusting them, testing each result. It’s impossible to test all combinations. I have some presets that worked for me on my remerger tool preset file, but I think it all depends on your concept.
- a. Text Encoder (base) Adjustment: Especially for SDXL the TE training often was responsible for some horrible interferences on the image. I have sometimes completely removed it and very often reduced it to 0.15 or 0.25. I have not tested this, but the same way with blocks, training the encoder and then removing it versus training without the encoder training might yell very different results. I can’t say what it’s the best practice here. I do think the option to train with the encoder for just 10% would be great, but it never worked in kohya.
- Rescale the LoRA StrengthAnyway, that it. Thanks for reading. Hope more people consider doing block weights for LoRAs!
- Finally test again the LoRA, consider that it might need a higher strength now.
- This is unnecessary but hardcoding a 1.4 LoRA to 0.8 might be beneficial if sharing on Civitai, as users often default to 1.0 or 0.8 without reading the settings.
- I haven’t tried this with Flux yet, so I’m not sure if my remerger tool can do it. For example, setting “all0.8” might replicate the effect achieved by AIToolkit’s rescaler for SD15 and SDXL. I haven’t tested Ostris’ script for Flux, but I imagine it doesn’t work.
Conectar