[api设置] What models are available for the Batch API? How does it work?

How does the Batch API work?

The Batch API endpoint allows users to submit asynchronous batch requests. OpenAI will process these requests within 24 hours. The details of each request will be read from a pre-uploaded file, and responses will be written to an output file. You can query the batch object for status updates and results.

What is the pricing for the Batch API?

Compared with the synchronous API, pricing for each model is 50% lower. Below is more information about OpenAI's API pricing.

Which models can I batch?

The following models can be used for batch processing:

gpt-3.5-turbo
gpt-3.5-turbo-16k
gpt-4
gpt-4-32k
gpt-4-turbo-preview
gpt-4-vision-preview
gpt-4-turbo
gpt-3.5-turbo-0301
gpt-3.5-turbo-16k-0613
gpt-3.5-turbo-1106
gpt-3.5-turbo-0613
gpt-4-0314
gpt-4-turbo-2024-04-09
gpt-4-32k-0314
gpt-4-32k-0613
text-embedding-3-large
text-embedding-3-small
text-embedding-ada-002

What happens if the API fails to complete requests within the promised time?

OpenAI aims to receive all requests within 24 hours. If a batch of requests expires (i.e., cannot be completed within the SLA time window), the remaining work will be cancelled, and all completed work will be returned. Developers will be charged for all completed work.

What is the limit on the number of requests I can batch?

There is no fixed limit on the number of requests you can batch; however, each usage tier has an associated batch rate limit. Your batch rate limit includes the maximum number of input tokens you can queue at one time. You can find your rate limits here.

After batch requests are completed, the batch rate limit resets because input tokens are cleared. The limit depends on the global number of requests in the queue. If the Batch API queue processes your batch quickly, the batch rate limit resets more quickly.

Can I change the time required to receive a response from the Batch API?

OpenAI's currently specified time window is 24 hours. OpenAI cannot currently change this time period.

Does the Batch API support streaming?

No, the Batch API does not support streaming.

Does the Batch API support images?

Yes, the Batch API supports images.

I received an error message: “The URL provided in this request does not match the batch endpoint.” What should I do?

If you receive this error, it means your URL is incorrectly formatted for the Batch API endpoint. Please refer to the documentation here to ensure you are calling the endpoint correctly.

Does Batch API usage count toward my other rate limits?

Batch API rate limits are completely separate from existing limits.

What happens if a batch is cancelled?

If a batch is manually cancelled, all completed results will be returned. Developers will be charged for all completed work.

What happens if a batch of tasks expires?

If a batch of jobs expires (i.e., cannot be completed within the SLA time window), the remaining work will be cancelled, and all completed work will be returned. You only need to pay for completed work.

What states can I expect for batch jobs?

A batch job can have any of the following statuses:

Validating: The uploaded file is being validated before the batch starts.
Failed: The file did not pass the validation process.
In progress: The file has been successfully validated and the batch is in progress.
Finalizing: The batch job has completed and results are being prepared.
Completed: The batch job has completed and results are ready.
Expired: The batch could not be completed within the SLA time window.
Cancelling: Cancellation of the batch job has been initiated.
Cancelled: The batch job has been cancelled.

How can I check which batch jobs have been created?

You can use the API call specified here to get a list of all batch jobs.

Does this endpoint support zero data retention?

If you have enabled zero data retention for your organization, note that this endpoint does not support zero data retention.