The following procedure outlines the basic steps for creating a character embedding in Automatic1111 Stable Diffusion
The basic steps are as follows:
- Create about 20 512 x 512 images of the subject in portrait
- Images must contain different backgrounds and lighting
- Use images that have the character showing different emotions
- Avoid having any other characters in the image
- Ensure there are not any contrasting shadows on the character
- Place these into a folder and name them all the same i.e. myCharacter
- Doing this in Windows is simple. Just select all the files , press F2 and rename one file, the other files will be renamed as well with a prefix of (1), (2), (3) etc
- In Automatic1111 go to the Train tab
- Select the sub-tab create embedding
- Under name add the name of your character (this is also usually the keyword) Make it unique i.e. P3t3rPan
- Under initialization text put in the same keyword
- If you are attempting to make another embedding using the same keyword as a previous embedding then check the overwrite checkbox
- Select the sub-tab preprocessing images
- In the source directory add the path to the folder containing the images created in step 2
- Add a new folder. Call it something meaningful like export or processed
- Copy the path for this folder and add it to the field for destination directory
- Set width and height to 512
- Existing Captions Txt Action can be kept at ignore (unless you are reprocessing the same images)
- For the other checkboxes I would only check use BLIP for Caption which will create a small txt file with a description of the image contents
- Press process
- the images and corresponding txt prompt descriptions will now be added to the destination folder
- Select the sub-tab Train
- Select the embedding you want to train from the dropdown menu
- set the embedding learning rate to between 0.00001 and 0.00005 (0.00005 will complete the training quicker and in fewer steps but may fallover (out of memory)
- leave gradient clipping alone
- Leave batch size as 1
- Under the dataset directory add the path to the folder that was populated in step 5
- Keep the log directory as Textual Inversion
- Select a [prompt template file from the folder that you should find in your SD instance following the path stable-diffusion-webui\textual_inversion_templates\ in this folder there are templates for prompts, iuse one of these or create your own. This will be used in the training
- Set width and height to 512
- Set Max steps to 2000 to begin with
- The next two fields, save images to directory and save copy should be set to 100. This will create a test image every 100 steps which is good for seeing how the training is progressing
- Check save images with embedding in PNG chunks but leave the other check boxes
- Leave choose latent sampling method as once
- Don’t press train embedding yet , there are a couple more steps to do
- Change your Stable Diffusion checkpoint model to an SD 1.5 model or the embedding won’t work
- Go to settings and under the SD VAE dropdown, set this to None
- Now you can press TrainEmbedding
The process should take anywhere between 30mins to an hour or more depending on your graphics card.
If after 2000 steps the images you are getting don’t seem to be ad good as you want then add another 1000 steps and continue the training on the model. Repeat this until you are happy with the results.
If the images start coming out with lots of noise then you have overstrained your model. You can go back steps by taking a working embedding file from the folder:
stable-diffusion-webui\textual_inversion\2023-Mnth–Date\MyCharacterName\embeddings
i.e. myCharacterName-3300.pt and add this to the embeddings folder.
Then return back to the sub-train tag and repeat step 6 onwards using a much slower training speed i.e 0.000001