[模型微调] Data merging: how to handle it when the number of collected data samples is too large—a summary of various methods and tools

[baike]Necessity of Classification and Integration[/baike]
After the data collection process ends, the data must be organized. This includes two parts: classifying the data and integrating it. The reason for doing this is to facilitate management, as well as to address issues such as data reading limits and speed. Generally, models such as APIs and GPTs have quantity limits and size limits, and they require us to provide some guidance on and introduction to the data to make it easier for GPT to understand.
[baike]Data Classification Work[/baike]
The classification work is relatively simple. We should pay attention during the collection process: place different collected content in different folders. Just pay attention to this step.
[baike]Data Integration Work[/baike]

The integration work is not complicated either. For different content types, we have different solutions. These include articles/documents, tables, images, videos, and audio, and we need to use different integration methods for them. Below we introduce them separately. There are actually many methods. Here I will only recommend a few methods that I commonly use and that the moderator recommends. If you find them useful, you can save them; I will keep updating.

[baike]Methods for Integrating txt-Type Content[/baike]
Using Python, the most common way we collect website content is to save articles from different sections of the website into corresponding local folders, using the article titles as file names. However, this results in a large number of txt documents.

Method 1: Python Script
The integration steps are likewise carried out through a Python script. The specific code is as follows.

import os

def merge_text_files(folder_path, output_file):
    with open(output_file, 'w', encoding='utf-8') as outfile:
        for filename in os.listdir(folder_path):
            if filename.endswith('.txt'):
                file_path = os.path.join(folder_path, filename)
                with open(file_path, 'r', encoding='utf-8') as infile:
                    for line in infile:
                        outfile.write(line)

# 注意路径字符串前的 r，表示这是一个原始字符串
merge_text_files(r'D:\360MoveData\Users\Administrator\Desktop\python\seleresult\huizong', r'D:\360MoveData\Users\Administrator\Desktop\python\seleresult\openai.txt')

Please change the folder location and the name of the merged file to your own.
Note: Please note that GPTs support 10 documents and a data volume of up to 8000KB per document.

[baike]Methods for Integrating Images and PDFs[/baike]
We have already provided this tool in the beginner's gift pack. It is a universal free PDF tool that can perform almost any merging, splitting, format conversion, etc. The link includes detailed download and usage instructions. If needed, please click the link below to download.
Tool introduction and download address: https://www.niuboyi.com/thread-635-1-1.html