ROADMAP.md

November 4, 2025 ยท View on GitHub

Feature Roadmap

This document includes the roadmap for the Gaggle DuckDB extension. It outlines features to be implemented and their current status.

Important

This roadmap is a work in progress and is subject to change.

1. Kaggle API

  • Authentication
    • Set Kaggle API credentials programmatically.
    • Support environment variables for authentication (KAGGLE_USERNAME and KAGGLE_KEY).
    • Support reading credentials from ~/.kaggle/kaggle.json file.
  • Dataset Operations
    • Search for datasets on Kaggle.
    • Download datasets from Kaggle.
    • List files in a dataset.
    • Get dataset metadata.
    • Dataset version awareness and tracking.
    • Download specific dataset versions (version pinning).
    • Check for dataset updates.
    • Upload DuckDB tables to Kaggle.

2. Caching and Storage

  • Cache Management
    • Automatic caching of downloaded datasets.
    • Clear cache functionality.
    • Get cache information (size and storage location).
    • Set cache size limit.
  • Storage
    • Store datasets in configurable directory.
    • Support for cloud storage backends (S3, GCS, and Azure).

3. Data Integration

  • File Format Support
    • CSV and TSV file reading.
    • Parquet file reading.
    • JSON file reading.
    • Excel (XLSX) file reading.
  • Querying Datasets
    • Replacement scan for kaggle: URLs.
    • Virtual table support for lazy loading.

4. Performance and Concurrency

  • Concurrency Control
    • Thread-safe credential storage.
    • Thread-safe cache access.
    • Concurrent dataset downloads.
  • Network Optimization
    • Configurable HTTP timeouts.
    • Retry logic with backoff for failed requests.
  • Caching Strategy
    • Incremental cache updates.
    • Background cache synchronization.

5. Error Handling

  • Error Messages
    • Clear error messages for invalid credentials.
    • Clear error messages for missing datasets.
    • Clear error messages for NULL inputs.
    • Detailed error codes for programmatic error handling.
  • Resilience
    • Automatic retry on network failures.
    • Local-only mode for cached datasets (via GAGGLE_OFFLINE).
    • Graceful degradation when Kaggle API is unavailable.

6. Documentation and Distribution

  • Documentation
    • API reference (see docs/README.md).
    • Usage examples (see the files in docs/examples/).
    • Other documentation files like the list of errors (check out docs/ directory).
  • Testing
    • Unit tests for core (Rust) modules.
    • SQL integration tests (run in DuckDB shell).
    • End-to-end integration tests with mocked HTTP.
    • Performance benchmarks.
  • Distribution
    • Built binaries for Linux, macOS, and Windows; AMD64 and ARM64.
    • Submission to the DuckDB's community extensions repository.

7. Observability

  • Logging
    • Structured logging (configurable via GAGGLE_LOG_LEVEL environment variable).