Whisp Copilot Instructions

September 25, 2026 · View on GitHub

Note: This file is intended for both AI assistants (such as GitHub Copilot) and human contributors. It defines project-specific coding standards, architectural guidelines, and best practices to ensure consistency and quality in all code contributions.

Project Overview

Whisp ("What is in that plot?") is a Python package for forest monitoring and deforestation risk assessment using Google Earth Engine (GEE). It implements the "Convergence of Evidence" approach by analyzing multiple satellite datasets to assess plots for compliance with deforestation-related regulations like EUDR.

Key capabilities: Process GeoJSON geometries through GEE to extract zonal statistics from 50+ datasets covering tree cover, commodity plantations, and forest disturbances (before/after 2020), then apply risk algorithms for different commodities (coffee, cocoa, rubber, palm oil, soy, livestock, timber).

Deployment Ecosystem

CRITICAL: This package powers multiple production systems. Breaking changes impact:

Two user groups:

  1. Direct users: Run package in notebooks with own GEE credentials (smaller group)
  2. API consumers: Use Whisp API endpoints (larger group, multiple platforms)

Backward compatibility is essential - changes to column names, output structure, or risk assessment logic affect all downstream systems.

Architecture

Core Pipeline Flow

  1. Input: GeoJSON → stats.py converts to Earth Engine FeatureCollections
  2. Dataset Preparation: datasets.py functions (suffix _prep) prepare GEE Image/ImageCollection objects
  3. Dataset Combination: datasets.py::combine_datasets() orchestrates dataset filtering and merging
  4. Statistics Extraction: stats.py::whisp_formatted_stats_geojson_to_df() runs zonal stats via reduceRegions()
  5. Risk Assessment: risk.py::whisp_risk() applies decision tree logic to generate risk columns

Configuration-Driven Design

  • lookup_datasets.csv defines ALL datasets, context columns, and metadata used in Whisp:
    • corresponding_variable column documents which function provides each dataset (for comprehension only, not used in code)
    • Controls which datasets feed into risk calculations: use_for_risk_pcrop (perennial crops) and use_for_risk_acrop (annual crops), which are currently held identical because the perennial/annual split is not yet wired (together they act as one combined crop-risk flag); plus use_for_risk_timber (timber)
    • Defines themes: treecover, commodities, disturbance_before, disturbance_after, context_and_metadata
    • National datasets use ISO2 codes in ISO2_code column; global datasets leave blank
  • config_runtime.py defines output column names and formatting rules
  • Schema validation via pd_schemas.py using Pandera

Critical Patterns

Dataset Function Naming Convention

Functions in datasets.py MUST follow strict naming:

  • Suffix: _prep (e.g., g_jrc_gfc_2020_prep)
  • Prefix: g_ for global datasets, nXX_ for national (XX = ISO2 code, e.g., nCI_bnetd_cocoa_prep for Côte d'Ivoire)
  • Return: ee.Image with .rename('DatasetName') matching CSV name column

Example from CSV → Function mapping:

name,corresponding_variable
EUFO_2020,g_jrc_gfc_2020_prep
Cocoa_bnetd,nCI_bnetd_cocoa_prep

Earth Engine Best Practices

  • Avoid .getInfo(): Keep operations server-side until final convert_ee_to_df() call
  • No loops over features: Use map() and reduceRegions() for batch processing
  • Cache expensive images: See get_water_flag_image() and get_admin_boundaries_fc() in stats.py - reuses global datasets across all features instead of recreating per-feature
  • Date filtering: Use module-level CURRENT_YEAR constant (from datetime.now().year) to avoid repeated calls

Unit Handling System

Whisp supports both hectares and percent units:

  • Unit type stored in column defined by stats_unit_type_column (default: "Unit")
  • risk.py::detect_unit_type() auto-detects or accepts explicit_unit_type override
  • All rows in a DataFrame must use same unit type (no mixing)
  • Risk thresholds (e.g., ind_1_pcent_threshold) are percentage thresholds regardless of unit type. Default is 10 for all indicators except ind_3_pcent_threshold (disturbance before 2020), which defaults to 50 because a "yes" there leads to a low risk outcome in the perennial crop tree

Risk Assessment Logic

risk.py::whisp_risk() implements commodity-specific decision trees:

  • Perennial crops (coffee, cocoa, rubber, palm): Uses Risk_PCrop output
  • Annual crops (soy): Uses Risk_ACrop
  • Livestock: Uses Risk_Livestock (NB still not integrated in Whisp main as of Jan 2026)
  • Timber: Uses Risk_Timber (includes additional categories like primary forests, logging concessions)

Decision tree checks in order:

  1. Treecover in 2020? (Indicator 1)
  2. Commodity presence in 2020? (Indicator 2)
  3. Disturbance before 2020-12-31? (Indicator 3)
  4. Disturbance after 2020-12-31? (Indicator 4)

Output values: "High", "Low", "More info needed"

Development Workflows

Code Style Principles

CRITICAL - Maintain existing code patterns:

  • Keep it simple: Avoid unnecessary complexity or "clever" solutions
  • Match existing style: Don't introduce decorators, classes, or patterns not already used in the codebase
  • Functional over OOP: Whisp uses simple functions, not class hierarchies - maintain this approach (until a future refactor)
  • Limit AI fingerprints: Code should be indistinguishable from existing codebase in style and complexity

Examples of what NOT to do:

  • ❌ Adding decorators when rest of codebase uses plain functions
  • ❌ Creating abstract base classes when existing code uses simple functions
  • ❌ Refactoring simple if/else into complex patterns
  • ❌ Adding type classes (dataclasses, Pydantic models) where dict/DataFrame suffices

What TO do:

  • ✅ Use same function signature patterns as existing code
  • ✅ Keep logic in simple, readable functions like datasets.py
  • ✅ Match commenting style and verbosity level
  • ✅ Preserve existing code organization (no unnecessary restructuring)

Note on future refactoring: While the current functional style works well and should be maintained for consistency, comprehensive refactoring to cleaner patterns is possible as a future deliberate effort. However, any such refactoring should be:

  • Discussed and planned (not done piecemeal by AI)
  • Applied consistently across the entire codebase
  • Not mixed with feature development

Making Changes Safely

Before modifying core functionality:

  1. Check if change affects API contract (column names, output structure, risk values)
  2. Consider impact on downstream systems (QGIS plugin, dashboards, Whisp Map)
  3. Maintain backward compatibility or version changes appropriately
  4. Document breaking changes clearly for API consumers

Testing with GEE

# conftest.py calls init_ee(): needs PROJECT in .env and a local EE sign-in
# (`earthengine authenticate`); with no credentials it opens a browser flow mid-run.
# Set WHISP_CLEAR_EE_CREDS=1 to delete the credentials file after the run (off by default).
pytest  # Runs basic tests for Whisp stats)

# Key test: tests/helpers/test_assess_risk.py
# - Uses fixtures/geojson_example.geojson (~50 test geometries)
# - Tests full pipeline: GeoJSON → stats → risk assessment

Adding New Datasets

  1. Add function to datasets.py: Follow naming convention (g_*_prep or nXX_*_prep)
  2. Add row to lookup_datasets.csv:
    • Set corresponding_variable to function name
    • Set theme (treecover/commodities/disturbance_before/disturbance_after)
    • Set both use_for_risk_pcrop=1 and use_for_risk_acrop=1 (keep them equal) if dataset should feed into crop risk calculations; the perennial/annual split is not yet wired, so the two columns are held identical
    • Set use_for_risk_timber=1 if dataset should feed into timber risk calculations
    • Set ISO2_code if national dataset
  3. No code changes needed for dataset to appear in output - CSV drives everything!
  4. Update documentation files as needed (see Documentation References below)

Code Quality Tools

# Poetry for dependency management
poetry install

# Pre-commit hooks handle formatting (pinned black) and run pytest on every commit

pre-commit install  # Set up hooks

Formatting: the pre-commit black hook (pinned version) is what runs; let it reformat rather than running a different black by hand. Ruff is configured in pyproject.toml ([tool.ruff]: max complexity 10, line length 120, type hints, import sorting) but its hook is currently disabled.

Running Locally vs. Colab

  • Local: Use virtual environment (.venv); for development pip install -e . from a clone (see README "Developer setup"), for plain use pip install --pre openforis-whisp
  • Colab: See notebooks/Colab_whisp_geojson_to_csv.ipynb for authentication flow
  • SEPAL: Special virtual environment setup (see SEPAL docs)

Key Gotchas

Column Name Dependencies

Many column names are hardcoded in various parts of the codebase and external systems. Changing them can break functionality. Some critical columns: Indicator (e.g., ind_1_treecover_2020) and risk columns (e.g., risk_pcrop) are both hardcoded in src/openforis_whisp/risk.py.

Further standard column names are defined in src/openforis_whisp/parameters/lookup_datasets.csv (rows where theme == context_and_metadata). Two key columns are specifically named to help ensure compatibility with external systems (such as the EU TRACES platform for EUDR):

When adding datasets/columns:

Legacy vs. Modern Functions

stats.py:

  • Modern: whisp_formatted_stats_geojson_to_df() — main entry point that routes to appropriate mode:
    • mode="concurrent": Uses advanced_stats.py with high-volume EE endpoint for parallel processing
    • mode="sequential": Uses advanced_stats.py with standard EE endpoint for single-threaded processing
    • mode="local": Uses local_stats.py for privacy-preserving local processing (downloads GeoTIFFs, uses exactextract)
    • mode="legacy": Calls the original implementation for backward compatibility
  • Legacy: whisp_formatted_stats_geojson_to_df_legacy() — kept for backward compatibility, but will be removed in the future.

Always use modern functions for new code. Legacy function works correctly but lacks newer optimizations.

GeoJSON Geometry Handling

  • Input: GeoJSON features with Polygon/MultiPolygon geometries
  • Internal: Stored as string in geometry_column (default "geo") during processing
  • Conversion: data_conversion.py handles GeoJSON ↔ EE ↔ DataFrame transformations
  • Tracking geometry changes from conversions: as the conversion process to EE can change the geometries, the optional 'geometry_audit_trail' parameter in whisp_formatted_stats_geojson_to_df(), allows the user to retain the original geometry column (prior to conversion to EE) in the output for comparison.

Documentation References

Performance Considerations

  • Batch processing: Process features concurrently in multiple batches via reduceRegions() using the GEE high-volume endpoint. Implemented in advanced_stats.py::whisp_ee_stats_fc_to_df_concurrent()
  • Local processing: Privacy-preserving mode downloads GeoTIFFs and uses exactextract for local zonal statistics. Implemented in local_stats.py::whisp_stats_local(). Obscures exact plot locations by using shifted/extended bounding boxes.
  • Asset caching: Water mask and admin boundaries cached at module level in stats.py
  • Filtering: Country-based filtering in reformat.py::filter_lookup_by_country_codes() reduces the additional national datasets processed per request. Aim to automate this in future to be driven by the input GeoJSON plot locations.
  • Avoid temporal filters in loops: Use module constants like CURRENT_YEAR instead of datetime.now() calls