[Sora培训内容] How does Sora keep characters unchanged, and how can consistent characters be designed?

Maintaining character consistency is very difficult, but it can be done. In the era of text-to-image generation with Midjourney, Stable Diffusion, and ChatGPT DALL·E 3, we have already achieved it; we gained experience through extensive testing and experimentation.

Maintaining character consistency requires getting 3 core things right:

1. A complete character description
2. Making the bot use the correct description
3. Use references: original image, seed, genid, and noise

1. Design a complete character description
Friends who have seen Sora examples should be clear that the examples Sora currently gives use one-sentence prompts. For example, it says "a woman, an African woman," and nothing else. Literally, it is not hard to see that a description like "a woman" is highly random: she may be tall or short, and her skin may be black or white. After you finish generating one video, when you generate the next video, you will find that the character is different.

From the examples, we randomly selected a set of prompt text: a woman, purple cargo pants, hat, Antarctica, sunset



We tried changing one element and changing the scene: still a woman, purple cargo pants, hat, Mumbai, sunset. From Antarctica to Mumbai, we can see that the character is already different—the hat, clothes, hair color, facial features, and face shape are all different. At this point the character has changed a lot, so phrases like "a woman" and "purple cargo pants, hat" are complete, yet random.



We tried changing another element and swapping the clothes: a woman, green dress, sun hat, Mumbai, sunset



Change it again: a woman, jeans and a white T-shirt, sun hat, Mumbai, sunset



We can see that when we change only the clothing, the face is still different, and the skin color is also different. In short, this is not the same person. We noticed that Sora's test control panel has a clothing description: jeans and a white T-shirt. This is a relatively complete description. We tried continuously using this clothing description to generate videos and see the effect.






You will find that when details are not emphasized, the clothing almost shows no change, because jeans in real life are almost that color, and a white T-shirt is almost like that—but not 100%. With more testing, it will still change.



Research conclusion

We tried more random conditions and it went wrong: this time the character put on a jacket. If you do not specify it, it will change styles and add additional styles based on other conditions. Through the simple tests above, we reached a conclusion: the key issue in ensuring feature consistency is a complete feature description.

These are some of the experiences we have researched. There are 3 problems that need to be solved:

1. Design the character appearance you want
2. A complete character description
3. Keep the character consistent

Research results

We used OpenAI, and when their DALL·E 3 generates comics, it has already achieved this. And we set up a trick: when you use a character name, it will generate a character with the same appearance.



2. Make the bot use the correct description
Just when we were secretly delighted by our success, we found that with more use, the randomness is still very high. Through more research, we found that sometimes it does not use our description; the bot often tampers with our content.

Through a large number of experiments, we found a pattern: no matter which AI you interact with, it will tamper with your input. Its logic is: you input it > it receives it > it understands it > it implements it in its own way.

Here is a simple example:




This problem is relatively simple to understand. Now that we know the problem, we need a good way to lock down our prompt and prevent it from making any changes. But the process is still very complicated.

3. seed, gen_id, and noise
seed is an image's seed number; every image has its seed. genid is a more advanced usage. They are all unique and can represent a particular image.


Noise is even more complex; it is a more hidden technology. It is what exists in image generation to control changes in image style.


So to control image consistency well, many control elements are needed. Once we use them well, we can encapsulate them and give them a code name. Next time I mention this code name, it will know what combination of elements to use.

So how do you fully control and make good use of these image-control and video-control technologies? We have created a complete tutorial.



Course introduction: http://www.niubo1ai.com/thread-531-1-1.html

If you want to quickly master this technology and get started quickly, you can sign up for the course.