[API调用] Use the gpt-4-vision-preview model from gp4api for video interpretation, implementing a video commentary feature.

The role of the gpt-4-vision-preview model is image recognition. For video, a video is composed of multiple frames, and each frame is a picture, so as long as we extract the pictures from these frames at the right time and send them to the API, and give it a certain prompt, the API model can interpret them very easily. We can also use prompts when outputting content to specify the output format, so that we can get the content we want. I will put the code at the end; first I'll explain the logic.

First, look at the moderator's example



Libraries used in this example

import cv2
import requests
import base64
import time
import json
from PIL import Image
import io

Script logic

1. Extract frame images

Use cv2 to extract video frames as images. Here we can customize it, for example, extract once every 5 seconds, which can minimize the workload and token usage as much as possible.

2. Compress the images
Compressing images with cv2 includes: edge detection, feature extraction, and size adjustment.
This step is key, because the subsequent image transcoding will generate a large string; a single image may be hundreds of thousands of bytes, and our model can receive at most 10,000 tokens at a time. If the images are not compressed, they simply cannot be uploaded, and they will also consume a large amount of our tokens.

3. Convert images into 64base-encoded strings

After image processing is complete, use the base64 library to convert our processed images into base64 encoding.
This step needs to be handled carefully. The moderator recommends testing locally first, without adding the API yet, and first testing the length of the encoding. Generally, I recommend compressing each image to a few hundred to a few thousand bytes before submitting; otherwise, a large number of frame images in one video added together will consume a lot of your tokens, and even $10 for 1 million tokens won't last for many runs.


4. Set the prompt

Once you are satisfied with the encoded length, when we submit to the API we also need to attach a prompt for the image. We can tell the model what our video topic is and give it a title; this is required by the model, and the logic is the same as when we normally use GPT-4 for image analysis. After uploading an image, you need to add a prompt to give it correct guidance. Otherwise, it will analyze the encoding itself~

Prompt



Output value



This is a prompt the moderator just roughly wrote. I told the model what my topic was, then asked it what to do and in what form to output the content. Of course, experts who understand prompts know how to better require its focus and output method here; we won't go into too much interpretation in this demonstration.

5. Other requirements

With the previous setup, you can already get the content you want. If you are not satisfied with the result, optimize the prompt. After finally getting satisfactory content, we can also use speech APIs for narration. For example, use OpenAI's TTS API, or make this program into a web version and use the browser's built-in text-to-speech.

6. Complete code display

You need to adjust it according to your own requirements. There are relevant comments for custom adjustments, including the character limit, the duration and interval for extracting frames, the image processing size, and the prompt step.

import cv2
import requests
import base64
import time
import json
from PIL import Image
import io

def feature_extraction(frame):
    """对图像进行简单的边缘检测，提取特征。"""
    gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
    edges = cv2.Canny(gray, 100, 200)
    return cv2.cvtColor(edges, cv2.COLOR_GRAY2BGR)

def resize_image(frame, target_width=128):
    """调整图像尺寸以进一步减小文件大小。"""
    height, width = frame.shape[:2]
    scaling_factor = target_width / float(width)
    target_size = (target_width, int(height * scaling_factor))
    resized_frame = cv2.resize(frame, target_size, interpolation=cv2.INTER_AREA)
    return resized_frame

def convert_to_webp_and_encode(frame, quality=80):
    """将图像帧转换为WebP格式并编码为base64字符串。"""
    pil_image = Image.fromarray(cv2.cvtColor(frame, cv2.COLOR_BGR2RGB))
    buf = io.BytesIO()
    pil_image.save(buf, format="WEBP", quality=quality)
    return base64.b64encode(buf.getvalue()).decode('utf-8')

def process_video_frame(frame):
    """处理视频帧以提取特征、调整大小、转换格式并编码。"""
    features_frame = feature_extraction(frame)
    resized_frame = resize_image(features_frame)
    img_base64_str = convert_to_webp_and_encode(resized_frame)
    return img_base64_str

API_URL = "https://api.openai.com/v1/chat/completions"
API_KEY = "sk-proj-m7ccmZ8Mt1WRBSapA0L7T3BlbkFJ3iiiVN4Iad5sAQPcYory"
video_path = r"C:\Users\Administrator\Downloads\Video\1.mp4"

cap = cv2.VideoCapture(video_path)
if not cap.isOpened():
    print("无法打开视频文件")
    exit()

frame_rate = int(cap.get(cv2.CAP_PROP_FPS))
interval_seconds = 5
frame_interval = frame_rate * interval_seconds
max_duration_seconds = 1 * 60  # 1 分钟
max_frames = min(max_duration_seconds * frame_rate, int(cap.get(cv2.CAP_PROP_FRAME_COUNT)))
frame_count = 0
data_list = []

while frame_count < max_frames:
    cap.set(cv2.CAP_PROP_POS_FRAMES, frame_count)
    ret, frame = cap.read()
    if not ret:
        break
   
    img_base64_str = process_video_frame(frame)
    prompt = "这是一个视频《ai通过图像识别技术发现了医生都没发现的乳腺癌》中的部分帧请分析编码进行图像解码，然后分析表达的图片内容，把我给你的片段组合在一起，解说一下不用描述这是什么直接说可以发挥你的想象，风趣一点请使用中文"
    response = requests.post(
        API_URL,
        headers={
            "Authorization": f"Bearer {API_KEY}",
            "Content-Type": "application/json"
        },
        json={
            "model": "gpt-4-vision-preview",
            "messages": [
                {"role": "system", "content": prompt},
                {"role": "user", "content": img_base64_str}
            ],
            "max_tokens": 200
        }
    )

    if response.status_code == 200:
        output = response.json()["choices"][0]["message"]["content"]
        print(f"Frame {frame_count}: 解读结果：{output}")
        data_list.append({"frame": frame_count, "output": output})
    else:
        print("请求失败:", response.text)

    frame_count += frame_interval
    time.sleep(interval_seconds)

cap.release()

with open('output.json', 'w', encoding='utf-8') as f:
    json.dump(data_list, f, ensure_ascii=False, indent=4)

print("所有数据已保存到 JSON 文件中。")