CubScrape - YouTube to Steam Game Discovery

August 6, 2025 · View on GitHub

A tool that scrapes YouTube gaming channels to discover Steam games, fetches their metadata, and presents them in a filterable web interface.

Features

  • Multi-Platform Support: Steam, Itch.io, and CrazyGames link extraction and metadata
  • Multi-Channel Support: Process multiple YouTube channels independently
  • Interactive Game Resolution: Prompts for low confidence matches, handles missing/depublished games
  • Sale Indicators: Bulk Steam price refresh with discount badges and sale detection
  • Web Interface: Filterable, sortable game discovery with unified 0-100 rating scale

Setup

Prerequisites

  • Python 3.12+
  • uv (Python package manager)
  • Node.js 20+
  • npm (comes with Node.js)
  • direnv (recommended for automatic environment loading)

Installation

  1. Clone the repository

  2. Install Python dependencies:

    uv sync --extra dev

    This creates a virtual environment and installs all dependencies including development tools.

  3. Install JavaScript dependencies:

    npm install

    TypeScript Support: The frontend has been migrated to TypeScript for improved type safety and developer experience. All Vue components and utilities now use TypeScript with strict type checking.

  4. Configure channels and options:

    # Edit config.json to configure YouTube channels and scraper options

    Configuration Options:

    • channels: YouTube channels to scrape with URLs and enabled status
    • skip_steam_matching: List of game names to exclude from Steam matching (e.g., "League of Legends")

Processing Modes

Note: All commands should be prefixed with uv run to ensure proper environment activation.

Backfill Mode - Process a specific channel with full options:

uv run cubscrape backfill --channel idlecub --max-new 20
uv run cubscrape backfill --channel dextag --max-steam-updates 10
uv run cubscrape backfill --channel olexa --max-new 50

Note: Steam games now use age-based refresh intervals automatically:

  • Games < 30 days old: Daily refresh
  • Games < 365 days old: Weekly refresh
  • Games ≥ 365 days old: Monthly refresh

Cron Mode - Process recent videos from all enabled channels:

uv run cubscrape cron

Reprocess Mode - Reprocess existing videos to extract new game links:

uv run cubscrape reprocess --channel idlecub

Single App Mode - Fetch specific Steam game data:

uv run cubscrape single-app --app-id 123456

Steam Price Refresh - Bulk update Steam game prices with sale indicators:

uv run cubscrape steam-price-refresh --max-apps 1000 --batch-size 500
uv run cubscrape steam-price-refresh --currencies eur --dry-run  # Preview EUR updates
uv run cubscrape steam-price-refresh --currencies both          # Atomic EUR+USD updates

Data Quality Mode - Check data integrity and completeness:

uv run cubscrape data-quality
# Identifies missing Steam games referenced in videos

Game Inference Mode - Find games from video titles and resolve missing Steam games:

uv run cubscrape infer-games
# Uses YouTube's game detection when available (more reliable)
# Falls back to title parsing for game name extraction  
# Interactive prompts for low confidence matches
# Resolves missing/depublished Steam games

Data is saved to:

  • data/videos-{channel}.json - YouTube video data per channel
  • data/steam_games.json - Steam game metadata
  • data/other_games.json - Itch.io and CrazyGames metadata

Viewing the Web Interface

The web interface is built with Vue 3, Vite, and Tailwind CSS.

Development Mode

# Start dev server (automatically builds DB if needed)
npm run dev

# Or skip DB check and just run Vite
npm run dev:vite

Visit http://localhost:5173/

Production Build

# Build for production
npm run build

# Preview production build
npm run preview

Database Setup

The Vue app expects a SQLite database at /data/games.db:

# Build the SQLite database
uv run cubscrape build-db
# OR
npm run build-db

Other Commands

# Lint and format code
npm run lint
npm run lint:fix
npm run format
npm run format:check

# TypeScript type checking (added during TypeScript migration)
npx tsc --noEmit           # Check types without emitting files
npx vue-tsc --noEmit       # Vue component type checking

TypeScript Migration

The frontend codebase has been fully migrated to TypeScript for enhanced type safety and developer experience. Key aspects of the TypeScript implementation:

Type System Features

  • Strict Type Checking: Comprehensive type definitions for all data structures
  • Database Types: Auto-generated types from SQLite schema for type-safe database operations
  • Vue Integration: Full TypeScript support for Vue 3 components with Composition API
  • Configuration Types: Strongly typed configuration system with JSDoc documentation

Key Type Definitions

  • Database Schema: /src/types/database.ts - Generated types matching SQLite database structure
  • Vue Components: All .vue files use TypeScript with <script setup lang="ts">
  • Utilities: Type-safe utilities in /src/utils/ with proper error handling
  • Configuration: Centralized configuration in /src/config/ with comprehensive type annotations

Development Workflow

  • Type Checking: npx tsc --noEmit for TypeScript validation
  • Vue Types: npx vue-tsc --noEmit for Vue component type checking
  • IDE Support: Full IntelliSense and error detection in modern IDEs
  • Build Integration: Vite handles TypeScript compilation automatically

The migration maintains full backward compatibility while adding type safety that catches errors at compile time and improves code maintainability.

Data Structure

Key data structures:

videos-{channel}.json - YouTube video data per channel:

{
  "videos": {
    "video_id": {
      "video_id": "...",
      "title": "...",
      "description": "...",
      "published_at": "...",
      "thumbnail": "...",
      "steam_app_id": "12345",
      "itch_url": "https://...",
      "crazygames_url": "https://...",
      "broken_app_id": "789012",
      "inferred_game": true,
      "inference_reason": "missing_steam_game",
      "channel_id": "idlecub",
      "channel_name": "Idle Cub",
      "last_updated": "2025-01-07T..."
    }
  },
  "last_updated": "2025-01-07T..."
}

steam_games.json - Steam game data:

{
  "games": {
    "12345": {
      "steam_app_id": "12345",
      "name": "...",
      "tags": ["..."],
      "positive_review_percentage": 85,
      "review_summary": "Very Positive",
      "is_early_access": false,
      "has_demo": true,
      "last_updated": "2025-01-07T...",
      ...
    }
  },
  "last_updated": "2025-01-07T..."
}