Table of Contents
About The Project
shopee-crawler is a simple project to crawling shopee's shop and product information. Some common information can be crawled:
- shop:
- shopid
- follower_count
- shop_location
- name
- ...
- product: can crawl all shop's products by recursive function
- itemid
- shopid
- name
- view_count
- like_count
- status
- categories
- price
- description
- currency
- attributes such as orgin, material,...
- ...
- review: can crawl all product's reviews by recursive function
- reviewid
- itemid
- orderid
- rating
- comment
- images
- videos
- ...
Built With
Getting Started
To get a local copy up and running follow these simple example steps.
Prerequisites
Make sure that you had installed Node.js
- npm
npm install npm@latest -g
Installation
- Clone the repo
git clone https://github.com/hoangkimminh/shopee-crawler
- Install NPM packages
npm install
Usage
- Modify
shops_usernameinsideget_shops_details.jswith your own shop's names
const shops_username = [
"minastore_vn",
"friday.official",
"germe.vn",
"sliky",
"yinxx.vn",
];
- Run scripts to get shop details
node get_shops_details.js
- Start crawling all products and its reviews of each shop with
node index.js
Data of each shop will be saved in output/[shopname].json Make sure output folder created
License
Distributed under the MIT License. See LICENSE for more information.
3 May 18, 2022
4 May 3, 2022
14 Jan 3, 2023
111 Dec 15, 2022
90 Nov 14, 2022
2 Apr 5, 2022
7 Sep 27, 2022
24 Dec 18, 2022
35 Dec 27, 2022
6 Oct 21, 2022