[api设置] What is OpenAI's TOKEN, and how is it calculated?

What Is a Token and How Is It Calculated?

What Is a Token?
Tokens can be thought of as parts of words. Before the API processes a request, the input is broken down into multiple tokens. These tokens are not cut precisely at the beginning or end of words; they may include trailing spaces or even subwords. Here are some useful rules of thumb for token length:

· 1 token ≈ 4 English characters
· 1 token ≈ ¾ of a word
· 100 tokens ≈ 75 words
or
· 1-2 sentences ≈ 30 tokens
· 1 paragraph ≈ 100 tokens
· 1,500 words ≈ 2048 tokens

To further understand how tokens accumulate, consider the following examples:

· Wayne Gretzky's quote "You miss 100% of the shots you don't take" contains 11 tokens.
· OpenAI's charter contains 476 tokens.
· A copy of the U.S. Declaration of Independence contains 1,695 tokens.

How words are split into tokens also depends on the language. For example, the Spanish "Cómo estás" ("How are you") contains 5 tokens (10 characters in total). A higher token-to-character ratio may make non-English API implementations more expensive.

You can use OpenAI's interactive Token tool to further explore tokenization, count tokens, and see how text is split into tokens. Note that the exact tokenization process differs across models. Newer models such as GPT-3.5 and GPT-4 use a different tokenizer from earlier models and generate different tokens for the same input text.

Alternatively, if you want to tokenize text programmatically, you can use Tiktoken, a fast BPE tokenizer designed specifically for OpenAI models.

Token Limits
Depending on the model used, a request can use up to 128,000 tokens, which are shared between the prompt and the completion. Some models (such as GPT-4 Turbo) have different limits for input and output tokens.

There are often creative ways to work within the limits, such as compressing the prompt text, splitting the text into smaller fragments, and so on.

Token Pricing
The API offers multiple model types at different prices. Pricing for requests also differs by model. For detailed token pricing information, please refer here.

Exploring Tokens
The API processes words according to context in the corpus data. The model receives a prompt, converts the input into a list of tokens, processes the prompt, and converts the predicted tokens back into the words that OpenAI sees in the response.

The same word may generate different tokens in different contexts. The following are examples of the API generating 'token values' based on context:


In the first example, the token “2266” represents “ red” (with a trailing space).


The token “2296” represents “ Red” (with a leading space and beginning with a capital letter), which is different from the token “2266” for lowercase “ red”.


When “Red” appears at the beginning of a sentence, the generated token does not contain a leading space. The token “7738” is different from the word in the previous two examples.

Observations:
The lower the token number, the more frequently that token appears in the corpus:

In all 3 sentences, the period has the same token number (“13”). This is because, in the corpus, the use of periods is very similar across contexts.

When ‘red’ appears in different positions in the sentence, the generated token is also different:

· Lowercase in the middle of a sentence: ‘ red’ · (token: “2266”)
· Uppercase in the middle of a sentence: ‘ Red’ · (token: “2297”)
· Uppercase at the beginning of a sentence: ‘Red’ · (token: “7738”)