Blog post: https://ai.meta.com/blog/movie-gen-media-foundation-models-generative-ai-video/

Samples: https://ai.meta.com/research/movie-gen/

Paper: https://ai.meta.com/static-resource/movie-gen-research-paper

• 30B parameter model capable of generating videos and images
• Video resolution 768x768 or similar amount of pixels in different aspect ratio
• Video length 16sec
• Blog mentions "potential future release" whatever that means

This looks like an interesting approach to using LLMs to help generate prompt specific workflows for ComfyUI.

I’m doing some research about AI tooling and trying to understand what kind of users prefer online vs local generation

View Poll

As you're aware, when training the vanilla in-painting model using SD1.5, we can utilize the diffusers pipeline and scripts, which typically require at least 20 steps to achieve optimal image generation results. However, hyper-inpainting models, such as this one link, only need around 4–6 steps to produce the same results. Both the vanilla and hyper-inpainting models consume similar resources and memory. I would like to know if the training process for hyper models differs from that of vanilla models, and how I can fine-tune a vanilla in-painting model to generate images in fewer steps during the inference phase, similar to the hyper in-painting model.

Newbie here : I have a small problem with Stability Matrix: all models are being copied into their own subfolder by Forge. The same thing happened when I installed Foocus. Where can I set it so that the original models folder is used by all programs together?

Going straight to the point, I fixed the Prodigy main issue. With my fix, you can train the Unet and TEs for as long as you want without frying the TEs and undertraining the Unet. To use it, just get the code I submitted in a PR on Prodigy’s GitHub. I don’t know if they’ll accept it, so you’ll probably have to manually replace it in the venv.

https://github.com/konstmish/prodigy/pull/20

Edit: it's also possible to put a different LR in each network

About the loss modifier, I made it based on my limited knowledge of diffusion training and machine learning. It’s not perfect, it’s not the holy grail, but my trainings always turn out better when I use it.

Feel free to suggest ways to improve it.

For convenience, I replaced OneTrainer's min snr gamma function with my own, so all I need to do is activate msg and my function will take over.

https://github.com/sangoi-exe/sangoi-loss-function

I’m not going to post any examples here, but if anyone’s curious, I uploaded a training I did of my ugly face in the training results channel on the OT discord.

Edit:

To use the prodigy fix, get the prodigy.py here:

https://github.com/sangoi-exe/prodigy/tree/main/prodigyopt

and put it in this folder:

C:\your-trainer-folder\OneTrainer\venv\Lib\site-packages\prodigyopt\

That's it, all the settings in OT stay the same, unless you want to set different LRs for each network, because that's possible now.

To use my custom loss modifier, get the ModelSetupDiffusionLossMixin.py here:

https://github.com/sangoi-exe/sangoi-loss-function

and put it in this folder:

C:\your-trainer-folder\OneTrainer\modules\modelSetup\mixin

Then in the OT's UI, select MIN_SNR_GAMMA in the Loss Weight Function on training tab, and insert any positive value other than 0.

The value itself doesn't matter, it's just to get OT to trigger the conditionals to use the min snr gamma function, which now has my function in place.

There was a typo in the function name in the loss modifier file, I fixed it now, it was missing an underline in the name.

04/10/2024: there was another typo inside the function 😅