了解AI 玩转AI Data Scraping Tool: Gpt-crawler, a General-Purpose Copy Crawler for Scraping Data from Websites—Installation and Usage Tutorial, a Beginner-Friendly Tool

A crawler is essential software for every AI enthusiast. No matter what kind of AI you work on, you cannot do without data. Gpt-crawler is highly recommended by the moderator and is beginner-friendly. Installation is simple; you only need to spend about 10-plus minutes before you can start your crawling tasks. Let's begin today's tutorial.

[baike]Install NOde.JS[/baike]Open the official Node website, https://nodejs.org/en. Click Download; after downloading, just click Next, Next.



[baike]Install Gpt-crawler[/baike]On the computer drive where you want to install it, create a folder, for example name it Gpt-crawler, then type CMD in the folder's path/address bar.



Enter the installation command to install the main program: git clone https://github.com/builderio/gpt-crawler



After installation, enter this folder and type cd gpt-crawler



After entering this folder, enter the dependency installation command: npm i  
At this point, installation is complete.

[baike]Configure the crawling file[/baike]
In the main program folder, click config.ts and open it with Notepad or any editing software.



To make it beginner-friendly, I opened it with Notepad so it is clearer. In it, I have already written the necessary parameters and the corresponding Chinese explanations. These configurations represent the URL we crawl and what the crawled content includes.


I have put the code here; you can copy it if needed.

import { Config } from "./src/config";


export const defaultConfig: Config = {
  url: "https://www.niuboyi.com/thread-222-1-1.html",
  /** URL to start crawling from; if a sitemap is provided, it will be used and all pages in the sitemap will be downloaded */
  match: "https://www.niuboyi.com/thread-222-1-1.html",
  /** Selector to get the inner text from */
  selector: `.docs-builder-container`,
  /** Content tags to crawl */
  maxPagesToCrawl: 50,
  /** Do not crawl more than this number of pages */
  outputFileName: "output.json",
  /** Output name */
  maxTokens: 2000000,
  /** Word count */
};

After configuration, enter the command npm start and press Enter to start crawling.


The crawled files are all in the output-1.json file in the directory. When this file has enough content, it can become your knowledge base.



[baike]Use the knowledge base to create a custom GPTS[/baike]

Open the GPTs creation page, https://chat.openai.com/gpts/editor



At this point, you now have your own crawler and can crawl the vast majority of websites