[隐私政策] How are ChatGPT and other language models developed?

OpenAI's large language models, including the models that power ChatGPT, were developed using three main sources of information:

(1) Information publicly available on the internet
(2) Information licensed from third parties
(3) Information provided by users or human trainers

This article outlines the publicly available information used to develop the models and how that information is collected and used in compliance with privacy laws. To learn how user information from the services is collected and used, including how to opt out of having ChatGPT conversations used to help train the models, please see the privacy policy and this Help Center article.

What is ChatGPT and how does it work?

ChatGPT is an AI-based service that you can access via the internet. You can use ChatGPT to organize or summarize text, or to write new text. ChatGPT is developed to understand and respond to users' questions and instructions. It does this by “reading” a large amount of existing text and learning how words appear in context. It then uses what it has learned to predict the most likely next word, and each subsequent word, that may appear in response to a user request. This is similar to the autocomplete feature on search engines, smartphones, and email programs.

For example, during the model's learning process (called “training”), the model might be asked to try to complete a sentence such as: “instead of turning left, she turned ___.” Before training, the model would answer with random words, but as it reads and learns from many lines of text, it will better understand this type of sentence and be able to predict the next word more accurately. It then repeats this process across a very large number of sentences.

Because many possible words could appear in this sentence (for example, instead of turning left, she turned “right,” “around,” or “back”), the model's responses have a certain degree of randomness, and in many cases the model will answer the same question in different ways.

A machine learning model is made up of a large number of numbers, called “weights” or “parameters,” together with code that interprets and executes these numbers. The model does not contain or store copies of the information it learns. Instead, as the model learns, some of the numbers that make up the model change slightly to reflect what it has learned. In the example above, the model read information that helped it improve from predicting random incorrect words to predicting more accurate words, but what actually happens in the model itself is just that the numbers change slightly. The model does not store or copy the sentences it reads.

Types of information used to teach ChatGPT

As described above, ChatGPT and other services are developed using

(1) information publicly available on the internet
(2) information licensed from third parties
(3) information provided by users or human trainers.

This article focuses on the first category of information: information publicly available on the internet.

For this set of information, only information that is free and publicly available on the internet is used—for example, it does not seek out paid information or information from the “dark web.” Filters are applied to remove information that is undesirable for the model to learn from or output, such as hate speech, adult content, websites that primarily aggregate personal information, and spam. This information is then used to teach the models.

As mentioned earlier, ChatGPT does not copy or store training information in a database. It actually learns the associations between words, and this learning helps the model