(Intermediate level of difficulty)
If you have read the previous post about online services to use AnimateDiff and create short animations videos or GIFs by just writting a couple of sentences. You have probably noticed that these links redirects to differents kind of interfaces and functions.
Some of these links may use ControlNet functions and some other AnimateDiff-cli-prompt-travel functions.For example, this link offers "prompt travel functions".
The main ability of "prompt travel" is to be able to change yourself the content of the prompt of the Nth frame. AnimateDiff-cli-prompt-travel offers mainly this function. The main problem with AnimateDiff is that it is a Command Line Interface and that it needs a json file to use it. Therefore it is not so user friendly. On the paper, DiffEx makes just AnimateDiff-cli-prompt-trave more convenient to use, but i think a visual interface makes totally sense to create visual content.
It is perfectly possible to use AnimateDiff-cli-prompt-travel with Automatic1111, it is in the same family of softwares and there is a plugin for that. But the interface that you have there is then, not very convenient.
Let's Install DiffEx v1.5
Installation and introduction for DiffEx v1.5
(Before you install make sure to have a good graphic card with probably at least 12GB of memory. This value is just to give you an idea.)
Unzip the zip file.
Make sure that the paths you are going to have with the software don't content any space.
You DON'T want a path like: "D:\test\DiffEx v1.5" for example.
This is to prevent a possible bug that I got in a test version (i can't tell if it is still actual).
Personally, it is installed here:D:\test\diffexDesktopUIFor_v15
Unzip the files where you are allowed to read, write and execute files.
Please read the file ReadMe.pdf included, especially but not limited, if you experience any problem with your installation.
It says that you need to have Python (3.10.11) and python in your path.
You can read the beginning of this other article, if you need some help to do this.
If you open a command prompt (CMD) in a random directory (not the directory of Python) and type:
python --version
It should return the version of Python. If not, don't try to go further but fix this before.
Double click animatediff-installer.bat to install "AnimateDiff CLI prompt"
(There is no need to move the bat file elsewhere and you don't need to be admin. Actually it seems actually better that you don't move the Bat file.)
This downloads "animatediff-cli-prompt-travel" and the packets that you need.
Wait that it is done.
If you have a Nvidia graphic card like me, Cuda has to be installed.
(For other cards, ask the manufacturer of your graphic card).
So for Nvidia type in a command prompt (CMD):nvcc --version
For me it returns something like 12.1 with more details
With me, this works fine, but the author of the software recommends to have the version 11.8 of CUDA.
So if this command returns an error for you, like the tool is not here, it is the version to install:
https://developer.nvidia.com/cuda-11-8-0-download-archive
You also need to install FFMpeg, this is needed to link the pictures together.
This is the download page of FFMpeg:
https://ffmpeg.org/download.html
FFMpeg needs probably also to be in your PATH too (do like for Python, but you have to adapt it to where you saved FFMpeg).
If FFMpeg is not installed correctly, you will get some kind of "FFProbe", later. To know if FFMpeg is installed correctly, open a command prompt in a random directory (not the one from FFMpeg) and type:
ffmpeg --help
If it doesn't return the help message of FFMpeg, FFMpeg is not installed correctly.
Directories:
After that, run DiffEx as admin to LINK correctly the directories.
Click on "LINK" in the software.
For AnimateDiff, you have to select the "animatediff-cli-prompt-travel" folder.
With me it is here:D:\test\diffexDesktopUIFor_v15\animatediff-cli-prompt-travel
This directory will be used to locate the motion modules:diffexDesktopUIFor_v15\animatediff-cli-prompt-travel\data\models\motion-module
If the software returns an error, just click "continue".
Just make sure all directories that you entered exist when you click "apply".
I think that the selection of the other 3 folders (models, vae and Lora) is optional.
However, even with the default settings, if you want to use VAE (optional), you have to create the "vae" directory in:
"data\models\vae"
If you have a folder with sd models (like a folder of automatic1111), you can select it for Models.
I personally like more to have a separated folder, so with MY settings,
I used the folder:D:\test\diffexDesktopUIFor_v15\animatediff-cli-prompt-travel\data\models\sd
(which is a probably useless choice, but I can't unsave it).
Likewise you could create (you NEED to create it) a VAE folder in:D:\test\diffexDesktopUIFor_v15\animatediff-cli-prompt-travel\data\models
and put some vae files inD:\test\diffexDesktopUIFor_v15\animatediff-cli-prompt-travel\data\models\vae
But the software can work without VAE, if your model is fine without.
If you need a VAE file, try this one:
https://huggingface.co/stabilityai/sd-vae-ft-mse-original/blob/main/vae-ft-mse-840000-ema-pruned.safetensors
and same thing for the Lora.
Note that with animatediff-cli-prompt-travel, you have 2 kinds of Lora.
Normal Lora (to help to render the pictures) and Motion Lora, to help to render the movement.
The embeddings are in:animatediff-cli-prompt-travel\data\embeddings
I like this one personnally that saves time from negative prompts:
easynegative.pt
that you can download here:
https://huggingface.co/datasets/gsdf/EasyNegative/tree/main
Please refer to the page 1 of the ReadMe.pdf to see what is the expected location of the files.
Note that you may have to create some of the directories yourself.data/models/motion_lora is one of them (root: the AnimateDiff folder)
You find motion Lora here
Now you need to make sure that you have all the models that you need.
Models:
1- You need first a motion module.
You can download some here:
https://huggingface.co/guoyww/animatediff/tree/main
The type of motion module that you have to use is function of the model that you use to generate the pictures.
If you generate pictures with SD 1.5 models, I suggest to use a mm_sd_v15_v2.ckpt
Put it in animatediff-cli-prompt-travel\data\models\motion-module
For SDXL you can try mm_sdxl_v10_beta.ckpt, but it doesn't work well.
You could also consider one of the models of this page:
https://civitai.com/tag/motion
(not all the models listed are motion modules, you should filter the results)
2- The model to create pictures:
Is it strongly recommended to use a SD 1.5 model.
CivitAi or huggingface are full of those.
mistoonAnime_v20.safetensors works decently.
If you linked a special folder for the models, make sure to put the model there, if not it is inanimatediff-cli-prompt-travel\data\models\sd
If your character is going to dance, make sure the model can actually reproduce dance movements.
An anime model trained on fighting postures is probably very capable.
3- The VAE (see above) this one is optional anyway.
---
For the moment you can't reach the other menus, so let's create the first steps of our first generation with DiffEx v1.5
At name, type: testdif
Model: mistoonAnime_v20
Motion module: mm_sd_v15_v2
Vae: optional
Scheduler: euler_a
Seed: -1
Steps: 20
CFG: 10
FPS: 8
You can replace the seed with 2569360667827952747 if you want
hires and LCM disabled
Those are my usual test settings.
Click on "save"
---
Description of the other menus of DiffEx:
Now you can access the other menus.
- Stylize uses a video to guess its prompts frame by frame.
- Prompt is the keyword that you use to generate the content.
So go to the prompt menu: and for head prompt type:
hyperrealism, Sphinx, wings, Photographic, cinematic photo, film, professional, 8k, highly detailed, golden jewelry, tanned, gorgeous, speaking, friendly, adult woman, riddle, mysterious, ancient Egypt, treasure hunt, silky clothes, golden bikini, magnificent, indoors, temple,
Tail prompt: exotic dancer
For negative prompt, type:
easynegative, ugly face, nude, nsfw
- Region is used to locate something on the picture.
A region mask can be a black and white picture or video.
All frames have to be black AND white.
It bugs if not.
White is where is the subject.
- Controlnet is used with pictures or videos
- Lora: it lists all the Loras, the normal ones and those that are related to movement. (but these 2 kind of Lora have to be in separated folders).
- For the advanced menu, just notice the context schedule.
The uniform option is supposed to be less stable as the composite one.
Again check the Readme file.
Let's keep it like that.
- For execute, make a test with:
512 and 512
fps: 12
length: 36
Format: gif
Context 16
(Usually, you don't change the Context).
Now press on "build and run" and wait.
The output is somewhere in your output directory.
So this is the result that i get, and what i showed you in this tutorial is the most basic thing you can do with DiffEx.
Usually, the only problem that you get with the output is the presence of "morphing" effect: one element is changed to something else.
You have many more things that you can do with DiffEx, so consider this as an introdution tutorial, so stay tuned.