Conectar
reForge WebUI updates, and: in need of a bit of help to support newer models, they are in the backend already!
Hi there guys, hope is all going good!
First of all, let me start by thanking you all guys for the great vibes and such for this project, it really helps me a lot to motivate to do things!
So, when I was doing some edits in the code, I did notice that most of the backend (about 80%) of Forge is yep, ComfyUI. They share the same files, code and such.
So thanks to Comfyanonymous and lllyasviel, Forge was possible.
With reForge now, I have decided to take 2 ways.
First, related to branches and such. I said I will update to all the upstream changes (I can do) up to A1111, and I will keep doing it.
But since I noticed this mix of A1111-Comfy backend, I will update as well to upstream Comfy changes.
These branches will be:
- main: Upstream A1111 changes.
- dev_upstream: Upstream A1111 and Comfy changes.
What does this mean?
- main branch will have the updates that come regularly to A1111, be dev or master. This will be mostly about:
- First, UI changes, like related to javascript, html and gradio that comes into the A1111 WebUI.
- Also samplers changes (like DDIM CFG++, PLMS CFG++, etc)
- New schedulers
- Denoiser settings
- Some Optimizations.
- All features itself that the WebUI implements, like skip Early CFG, refiner, etc as we know on the WebUI.
- More!
- dev_upstream will have the updates that come regularly to A1111, be dev or master and upstream changes (when applicable as A1111) from the Comfy backend. This mean:
- Most of the optimizations.
- New CLIPs.
- New model_management, model_loader, VRAM features.
- Support for new models.
- New Samplers and Schedulers (when applicable).
- New features that can be applied from args.
- Fixes for LoRAs, DoRAs, Lycos, GLoRAs and more.
- Support more extra networks.
- More!
Okay, after all that, the main goal of the post.
-------------
Forge uses a file called forge_loader.py to actually load the models and start to doing inference, located at modules_forge folder. This file actually handles SD1.5, SD2.x and SDXL models.
I have (mostly) added code in the backend (that come from Comfy, so again, really thanks to him!), related for these next models:
- CosXL
- SD Cascade
- SD3
- Koala 700M/1B
- SV3D
- Stable Audio
- AuraFlow
The code to make these models work are there, in reForge! But, since they aren't managed on forge_loader, they will actually fail to load by different errors.
So I wanted to ask, if possible, help to try to implement these models!
For the Audio or Video models, that can't be directly converted possibly, but for the rest, assuming you can load a single checkpoint it can work fairly directly. Special case is Cascade where you have to load various checkpoints at the same time, so a UI modification would be needed.
------
Also, besides that, I think the controlnet built it extension *could be* updated as how it manages Comfy backend, but that is something I have to check; for sure it won't be easy.
------
As for a TL:DR:
- Branch main has A1111 upstream changes.
- Branch dev_upstream has A1111 upstream and Comfy upstream changes.
- Backend code for new models are on dev_upstream branch, but have to implement them on forge_loader.py to work.
So that's all guys. Time to sleep, I have been like 24 hours awake making commits and I think I'm bit tired.
Take care!
[link] [comments]
