[数据分析] OpenAI releases the CriticGPT model to check errors in ChatGPT-generated code and shows you how to use it for error correction.

CriticGPT is a GPT-4-based model that critiques ChatGPT’s responses to help human trainers spot errors during RLHF.

[baike]The current error problem[/baike]OpenAI trained a GPT-4-based model called CriticGPT to find errors in ChatGPT’s code output. OpenAI found that when reviewing code with CriticGPT’s help, its performance was 60% better than without it. OpenAI is working to integrate models like CriticGPT into OpenAI’s RLHF labeling pipeline, giving OpenAI’s trainers explicit AI assistance. This is a step toward being able to evaluate the outputs of advanced AI systems, which people may struggle to rate without better tools.

[baike]Thinking about how to solve the problem[/baike]The GPT-4 family of models that powers ChatGPT achieves usefulness and interactivity through “reinforcement learning from human feedback” (RLHF). A key part of RLHF is collecting comparisons, in which people known as AI trainers rate different ChatGPT responses against one another.

As OpenAI makes progress in reasoning and model behavior, ChatGPT becomes more accurate and its errors become more subtle. This can make errors hard for AI trainers to spot, making comparison tasks that support RLHF more difficult. This is a fundamental limitation of RLHF, and as models gradually become more knowledgeable than anyone who can provide feedback, it may make aligning models increasingly difficult.

To address this, OpenAI trained CriticGPT to do review work and highlight inaccuracies in ChatGPT’s answers.



CriticGPT’s suggestions are not always correct, but OpenAI found that, compared with no AI help, they can help trainers find more problems in model-written answers. In addition, when using CriticGPT, AI strengthens their capabilities, producing more thorough reviews than people alone and fewer errors than the model working by itself. In OpenAI’s experiments, a second trainer preferred reviews from the Human+CriticGPT team over reviews from unassisted humans more than 60% of the time.



[baike]How to use it[/baike]Like ChatGPT, CriticGPT is trained using RLHF. But unlike ChatGPT, it sees large numbers of inputs containing errors and must then review those errors. OpenAI asked AI trainers to manually insert these errors into code written by ChatGPT and then write sample feedback as if they had found the errors just inserted. The same person then compared multiple reviews of the modified code, so they could easily judge whether a review had found the errors they inserted. In OpenAI’s experiments, OpenAI studied whether CriticGPT could catch inserted errors as well as “naturally occurring” ChatGPT errors found by previous trainers. OpenAI found that in 63% of naturally occurring error cases, trainers preferred CriticGPT’s review over ChatGPT’s review, partly because the new reviewer produced fewer “nitpicks” (unhelpful minor complaints) and hallucinated fewer problems.

OpenAI also found that by performing additional test-time search over the review reward model, OpenAI could generate longer, more comprehensive reviews. This search process allowed OpenAI to balance how aggressively OpenAI looks for problems in code and to configure a precision-recall tradeoff between hallucinations and the number of detected errors. This means OpenAI can generate reviews that are as helpful as possible for RLHF.

[baike]Technical limitations[/baike]OpenAI trained CriticGPT on short answers from ChatGPT. To supervise future agents, OpenAI needs to develop methods that can help trainers understand long and complex tasks.

Models still hallucinate, and sometimes trainers make labeling errors after seeing these hallucinations.

Sometimes real-world errors can be spread across many parts of an answer. OpenAI’s work focuses on errors that can be pointed to in one place, but in the future OpenAI will also need to address scattered errors. The help CriticGPT can provide is limited: if a task or response is extremely complex, even human experts assisted by a model may not be able to evaluate it accurately.

[baike]Upgrade plan[/baike]To adapt to increasingly complex AI systems, OpenAI needs better tools. In OpenAI’s CriticGPT research, OpenAI found that applying RLHF to GPT-4 is expected to help humans generate better RLHF data for GPT-4. OpenAI plans to further scale this work and put it into practice.