README.md

  __  __      _____
 |  \/  |    |  __ \
 | \  / |    | |  | |
 | |\/| |    | |  | |
 | |  | |    | |__| |
 |_|  |_|    |_____/

🚀 Overview

A production-ready CLI scraping system built with Python. It collects, cleans, and stores product data from multiple e-commerce platforms, providing analysis-ready datasets in multiple formats.

• Multi-Platform: Amazon, Noon, AliExpress, Jumia, and eBay.
• High Volume: Support for up to 1000 pages per site (~20,000 products).
• Modern UI: Minimalist "Claude Code" inspired CLI with a clean Peach-theme.
• Clean Data: Automatic validation, deduplication, and normalization.

📦 Platforms

⚡ Quickstart

1. Install Dependencies:

   pip install -r requirements.txt
   

2. Run Interactive Mode (Recommended):

   python main.py
   

3. Or use direct commands:

   # Scrape 10 pages from Amazon
   python main.py scrape amazon 10
   

💻 CLI Usage

🔍 Scraping

python main.py scrape all 5              # Scrape all sites (5 pages each)
python main.py scrape noon 10 -q "phone" # Scrape specific site with keyword
python main.py scrape ebay 5 -f csv      # Scrape and export to CSV directly

📊 Data & Analysis

python main.py analyze                   # Show full analysis report
python main.py stats                     # Show quick database statistics
python main.py sites                     # List all available platforms

📂 Exporting

python main.py export csv                # Export all stored data to CSV
python main.py export excel              # Export to Excel format

📁 Project Structure

md-scraper/
├── main.py              # Entry point (CLI + Interactive)
├── config/              # Configuration & Settings
├── src/
│   ├── scrapers/        # Scraping logic for all sites
│   ├── models/          # Data structures (Product, Price)
│   ├── pipeline/        # Cleaning & Normalization
│   ├── storage/         # Database & Exporters
│   └── utils/           # Colors, HTTP Client, Logger
├── data/                # Database & Exported files
└── logs/                # Application logs

⚙️ Configuration

Control the scraper behavior using environment variables or settings.py:

💾 Data & Storage

• Database: All scraped data is stored in data/scraper.db (SQLite).
• Exports: Found in data/processed/.
• Custom Sites: Add your own sites via the interactive menu; they are saved in data/custom_sites.json.

📖 Documentation

For more detailed information, check the module-specific documentation:

• Developer Guidelines
• Source Code Structure
• Scraper Development

⚖️ License & Ethical Use

• License: MIT - See LICENSE for details.
• Ethics: Please respect robots.txt and use the built-in delays to avoid overwhelming servers.