音乐生成 Interpretation of OpenAI's Powerful Voice Cloning Software Release and Application Scenarios

OpenAI is committed to developing safe AI that benefits broadly. Today, we are sharing initial insights and results from a small-scale preview of a model called “Voice Engine,” which uses text input and a single 15-second audio sample to generate natural speech that closely resembles the original speaker. Notably, a small model with a single 15-second sample can create emotionally rich and realistic voices.

We first developed Voice Engine in late 2022 and used it to power the text-to-speech API and the preset voices available in ChatGPT Voice and Read Aloud. At the same time, because of the potential for abuse of synthetic voices, we are taking a cautious and informed approach to broader release. We want to open a conversation about the responsible deployment of synthetic voices and how society can adapt to these new capabilities. Based on these conversations and the results of small-scale tests, we will make more informed decisions about whether and how to deploy this technology at scale.

Early Applications of Voice Engine
To better understand the potential uses of this technology, late last year we began testing it privately with a small group of trusted partners. We have been impressed by the applications this group has developed. These small-scale deployments help provide us with approaches, safeguards, and a way of thinking about how Voice Engine can be used effectively across industries. Some early examples include:

In What Scenarios Are Voice Cloning Applications Used?

Providing reading assistance to non-readers and children through natural, emotionally expressive voices that represent a broader range of speakers than preset voices. Age of Learning, an education technology company dedicated to children’s academic success, has been using it to generate pre-written voiceover content. They also use Voice Engine and GPT-4 to create real-time, personalized responses to interact with students. With this technology, Age of Learning is able to create more content for a broader audience.
​​​​
Translating content such as videos and podcasts so that creators and businesses can reach more people around the world fluently in their own voices. HeyGen is an early adopter of this technology, an AI visual storytelling platform that works with enterprise customers to create customized human-like avatars for a range of content, from product marketing to sales presentations. They use Voice Engine for video translation, so a speaker’s voice can be translated into multiple languages and reach global audiences. When used for translation, Voice Engine preserves the original speaker’s native accent: for example, using an audio sample from a French speaker to generate English will produce speech with a French accent.

Reaching communities around the world by improving the delivery of basic services in remote areas. Dimagi is building tools for community health workers to provide a range of basic services, such as counseling for breastfeeding mothers. To help these workers develop skills, Dimagi uses Voice Engine and GPT-4 to provide interactive feedback in each worker’s primary language (including Swahili) or in more informal language (such as the code-mixed language popular in Kenya).

Good nutrition is very important for ensuring children’s healthy physical and mental growth. Foods such as fruits, vegetables, protein, calcium, and various vitamins are very important for bone growth and brain development. Eating well means a child has a strong immune system capable of resisting disease. This means that even if influenza is spreading on the streets, the child will have a strong ability to cope. In this way, there is no need to take him to the hospital often. By doing this, we are cultivating a strong generation. As you know, the future of our society rests in the hands of these young people. We are delighted that we have given them the best start in life.
Supporting non-speaking people, for example by providing therapeutic applications for people with conditions that affect speech and educational enhancement services for people with learning needs. Livox is an AI alternative communication application that powers augmentative and alternative communication (AAC) devices, enabling people with disabilities to communicate. By using Voice Engine, they are able to provide people with unique, non-robotic voices for non-speaking individuals across multiple languages. Their users can choose the voice that best represents them, and for multilingual users, maintain a consistent voice in each spoken language.

Helping patients with sudden-onset or degenerative speech conditions restore their voices. Lifespan’s Norman Prince Neurosciences Institute, a nonprofit health system and the primary teaching affiliate of Brown University’s medical school, is exploring applications of AI in clinical settings. They have been piloting a project to provide Voice Engine to individuals with speech impairments of oncological or neurological origin. Because Voice Engine requires such a short audio sample, doctors Fatima Mirza, Rohaid Ali, and Konstantina Svokos used audio from a video recorded for school to restore the voice of a young patient who had lost fluent speech due to a vascular brain tumor.

Building Voice Engine Safely
We recognize that generating speech that resembles people’s voices poses serious risks, which are especially concerning in an election year. We are working with U.S. and international partners across government, media, entertainment, education, civil society, and other fields to ensure we incorporate their feedback as we build.

Partners testing Voice Engine today have agreed to our usage policies, which prohibit impersonating other individuals or organizations without consent or legal right. In addition, our terms with these partners require the explicit and informed consent of the original speaker, and we do not allow developers to create their own voices for individual users. Partners must also clearly disclose to audiences that the voices they hear are AI-generated. Finally, we have implemented a set of safety measures, including adding watermarks to trace the origin of any audio generated by Voice Engine and proactively monitoring how it is used.

We believe that any broad deployment of synthetic voice technology should be accompanied by a voice authentication experience to verify whether the original speaker intentionally added their voice to the service, as well as a prohibited voice list to detect and prevent the creation of voices that are too similar to prominent figures.

Looking Ahead

Voice Engine is a continuation of our commitment to understanding the frontiers of technology and sharing AI openly. In line with our approach to AI safety and our voluntary commitments, we are choosing to preview but not broadly release this technology at this time. We hope this preview of Voice Engine both highlights its potential and sparks the need to strengthen society’s resilience to the challenges posed by increasingly convincing generative models. Specifically, we encouragethe following measures:

Phasing out voice-based authentication as a security measure for accessing bank accounts and other sensitive information
Exploring policies to protect the use of individuals’ voices in AI
Educating the public about the capabilities and limitations of AI technology, including the possibility of deceptive AI content
Accelerating the development and adoption of technologies that trace the origin of audiovisual content, so that it is always clear whether you are interacting with a real person or AI
It is important that people around the world understand where this technology is headed, regardless of whether we ultimately deploy it broadly. We look forward to continuing the conversation with policymakers, researchers, developers, and creatives about the challenges and opportunities of synthetic voices.