📊 CIA JSON Export System - Static CDN Deployment

May 24, 2026 · View on GitHub

Comprehensive Political Intelligence Data for JavaScript Applications

Version: 1.1.0
Date: 2024-12-03
Purpose: Static JSON data exports for CDN hosting and JavaScript consumption
Update Frequency: Daily automated updates

✅ Schema Validation Status

Validate JSON Schemas

The 5 JSON export schemas are continuously validated against 138 relevant CSV sample data files (filtered from 210+ total files) to ensure correctness. See VALIDATION_README.md for details.

Latest Validation: 5 schemas validated, 138 files analyzed (View Report)
Test Coverage: 18 unit tests covering field parsing, type inference, and validation logic (Run Tests)

Validation Notes:

  • 125 field mismatches documented between original projected schemas and actual implemented data
  • Mismatches represent fields that were planned but not yet implemented in the database
  • Data-validated schemas (*-schema-validated.md) reflect actual current data structure (Summary)
  • Intelligence schema passes with 0 mismatches ✅

🔗 Business Context & Product Integration

These JSON specifications are the technical foundation for the CIA product portfolio, enabling all 6 product lines with structured political intelligence data.

Business Strategy Documentation

Market Opportunity

The JSON specifications power products serving 5 customer segments with €46M Total Addressable Market:

  • Political Consulting (€15M TAM) - Politician risk assessment, voting statistics
  • Media & Journalism (€8M TAM) - Real-time political monitoring, investigative data
  • Academic Research (€5M TAM) - Comprehensive datasets, bulk exports
  • Corporate Affairs (€12M TAM) - Government affairs monitoring, regulatory tracking
  • Financial Services (€20M+ TAM) - Political risk intelligence, early warning systems

Product Line Integration

Product LineJSON Schemas UsedRevenue PotentialCustomer Segments
Political Intelligence APIpolitician-schema.md, party-schema.md€630K/yearPolitical consulting, media, academic
Advanced Analytics Suiteintelligence-schema.md, All schemas€855K/yearCorporate affairs, NGOs, political parties
Risk Intelligence Feedintelligence-schema.md€1.77M/yearFinancial services, corporate risk, consulting
Predictive Analyticsintelligence-schema.md, party-schema.md€2.05M/yearStrategic consulting, corporate strategy, campaigns
White-Label PlatformAll schemas€1.7M-4.7M/yearGovernment agencies, international orgs, consulting
Decision Intelligenceintelligence-schema.md, ministry-schema.md€2.09M/yearLobbying, corporate affairs, media

Total Annual Revenue Potential: €9.1M+ from JSON-powered products

🎯 Executive Summary

The CIA JSON Export System transforms the comprehensive political intelligence database into structured, versioned JSON files optimized for static CDN hosting. This enables lightweight JavaScript applications, data visualization tools, and third-party integrations to access Swedish political data without direct database connections.

Key Benefits

  • 🚀 High Performance: Static JSON files served via CDN with global edge caching
  • 💰 Cost-Effective: Eliminate database query costs, pay only for storage and bandwidth
  • 🔒 Secure: Read-only data access, no database credentials exposed
  • 🌐 Universal Access: CORS-enabled, accessible from any JavaScript application
  • 📱 Mobile-Friendly: Optimized JSON sizes for mobile and low-bandwidth scenarios
  • 🔄 Always Current: Daily automated updates from live database

Data Coverage

Entity TypeRecordsUpdate FrequencyAvg File Size
Politicians~350 activeDaily~2.5 MB
Parties8 parliamentaryDaily~500 KB
Ministries12 governmentDaily~800 KB
Committees15 riksdagDaily~600 KB
Intelligence ProductsRisk scores, trendsDaily~1.2 MB

Implementation Status

✅ v1.36 Update (2024-12-08): Core JSON export service layer completed

Implemented Components:

  • ✅ IntelligenceExportService - Service interface for JSON exports (service.data.api)
  • ✅ IntelligenceExportServiceImpl - Jackson-based JSON serialization (service.data.impl)
  • ✅ Export DTOs - Data transfer objects for risk assessments, coalition alignment, temporal trends
  • ✅ Unit Tests - Comprehensive tests with Mockito (6/6 passing)
  • ✅ Build Integration - Clean compile and test pass in CI/CD pipeline

Ready for Integration:

  1. exportRiskAssessments() - Exports all rule violations with severity and metadata
  2. exportCoalitionAlignment() - Exports party alignment matrix with voting cohesion
  3. exportTemporalTrends() - Exports decision trends with daily/weekly/monthly moving averages
  4. writeJsonToFile() - File writer for CDN-ready static JSON generation

Next Steps:

  • REST endpoint integration (planned)
  • Scheduled daily export job (planned)
  • CDN deployment automation (planned)

Technical Foundation: Service layer built on Spring Framework with JPA/Hibernate, using existing DAO layer for database access to:

  • RuleViolationDAO → Risk assessment data (50 behavioral rules)
  • ViewRiksdagenCoalitionAlignmentMatrixDAO → Coalition alignment data
  • ViewDecisionTemporalTrendsDAO → Temporal trend analysis data

📋 Table of Contents

  1. Architecture Overview
  2. JSON Schema Specifications
  3. Data Categories & Labels
  4. Field Completeness Validation
  5. File Structure & Naming
  6. CDN Deployment Guide
  7. JavaScript Usage Examples
  8. Update & Versioning Strategy
  9. API Compatibility

🏗️ Architecture Overview

%%{
  init: {
    "theme": "base",
    "themeVariables": {
      "primaryColor": "#e8f5e9",
      "primaryTextColor": "#2e7d32",
      "lineColor": "#4caf50",
      "secondaryColor": "#e3f2fd",
      "tertiaryColor": "#fff3e0",
      "primaryBorderColor": "#2e7d32",
      "fontSize": "16px"
    }
  }
}%%
graph TB
    subgraph "📊 Data Source Layer"
        DB[(PostgreSQL Database<br/>85 Views)]
        VIEWS["Intelligence Views<br/>Politicians, Parties,<br/>Ministries, Committees"]
    end
    
    subgraph "🔄 Export Pipeline"
        EXPORT[JSON Export Service<br/>Java/Spring]
        TRANSFORM[Data Transformation<br/>Schema Mapping]
        VALIDATE[JSON Validation<br/>Schema Compliance]
    end
    
    subgraph "📦 Storage Layer"
        S3[S3 Bucket<br/>Static JSON Files]
        VERSIONS["Version Control<br/>v1, v2, latest"]
    end
    
    subgraph "🌐 CDN Distribution"
        CDN[CloudFront CDN<br/>Global Edge Locations]
        CACHE[Edge Caching<br/>24h TTL]
    end
    
    subgraph "👥 Consumer Applications"
        WEB[Web Applications<br/>JavaScript/React]
        VIZ[Data Visualizations<br/>D3.js/Chart.js]
        MOBILE[Mobile Apps<br/>Hybrid/PWA]
        THIRD[Third-Party APIs<br/>Integrations]
    end
    
    DB --> VIEWS
    VIEWS --> EXPORT
    EXPORT --> TRANSFORM
    TRANSFORM --> VALIDATE
    VALIDATE --> S3
    S3 --> VERSIONS
    VERSIONS --> CDN
    CDN --> CACHE
    CACHE --> WEB
    CACHE --> VIZ
    CACHE --> MOBILE
    CACHE --> THIRD
    
    style DB fill:#c8e6c9,stroke:#2e7d32,stroke-width:3px
    style EXPORT fill:#bbdefb,stroke:#1976d2,stroke-width:3px
    style S3 fill:#fff9c4,stroke:#f57f17,stroke-width:3px
    style CDN fill:#ffccbc,stroke:#d84315,stroke-width:3px
    style WEB fill:#e1bee7,stroke:#7b1fa2,stroke-width:3px

Data Flow Pipeline

  1. 📥 Data Extraction: Query materialized views from PostgreSQL database
  2. 🔄 Transformation: Map database records to JSON schema with labels
  3. ✅ Validation: Verify JSON schema compliance and data quality
  4. 💾 Storage: Write versioned JSON files to S3 bucket
  5. 🚀 Distribution: Invalidate CDN cache and propagate to edge locations
  6. 📊 Consumption: JavaScript applications fetch JSON via HTTPS

📄 JSON Schema Specifications

Schema Design Principles

  1. Consistent Structure: All JSON files follow standardized patterns
  2. Rich Metadata: Include timestamps, versions, source attribution
  3. Multi-Level Descriptions: Short (tweet), long (paragraph), detailed (essay)
  4. Category Labels: Hierarchical classification system
  5. Relationship Links: Cross-references between entities
  6. Performance Optimized: Minimal nesting, efficient serialization

Common JSON Structure

{
  "metadata": {
    "version": "1.0.0",
    "generated": "2024-11-24T02:23:58Z",
    "source": "Citizen Intelligence Agency",
    "schema": "politician-profile",
    "recordCount": 349,
    "dataDate": "2024-11-23"
  },
  "data": [
    {
      "id": "unique-identifier",
      "labels": {
        "category": "primary-classification",
        "subcategories": ["tag1", "tag2"],
        "descriptions": {
          "short": "One-line summary (max 140 chars)",
          "long": "Paragraph description (max 500 chars)",
          "detailed": "Comprehensive overview with context"
        }
      },
      "attributes": {
        // Entity-specific fields
      },
      "relationships": {
        // Cross-references to related entities
      },
      "intelligence": {
        // Analytical metrics and scores
      }
    }
  ]
}

Schema-to-Product Mapping

Each JSON schema directly supports specific product features and customer use cases:

Schema FileProduct LinesKey FeaturesBusiness ValueCustomer Segments
politician-schema.mdPolitical Intelligence API, Advanced Analytics, Risk IntelligencePolitician profiles, voting records, risk scores, activity metricsCore API product, €15M TAMPolitical consulting, media, academic
party-schema.mdPolitical Intelligence API, Advanced Analytics, Predictive AnalyticsParty performance, coalition patterns, voting trendsCoalition analysis, €12M TAMCorporate affairs, political consulting
committee-schema.mdAdvanced Analytics, Decision IntelligenceCommittee composition, proposal tracking, effectiveness metricsLegislative monitoring, €8M TAMGovernment affairs, lobbying
ministry-schema.mdDecision Intelligence, White-Label PlatformGovernment ministry data, decision flows, policy effectivenessRegulatory forecasting, €10M TAMCorporate risk, consulting firms
intelligence-schema.mdRisk Intelligence, Predictive Analytics, Decision IntelligenceRisk assessment, predictions, decision analytics, trend forecastingPremium analytics, €30M+ TAMFinancial services, strategic consulting

See Complete Product Documentation: BUSINESS_PRODUCT_DOCUMENT.md

Data Model Overview

%%{
  init: {
    "theme": "base",
    "themeVariables": {
      "primaryColor": "#e1f5ff",
      "primaryTextColor": "#0277bd",
      "lineColor": "#0288d1",
      "secondaryColor": "#c8e6c9",
      "tertiaryColor": "#fff9c4",
      "primaryBorderColor": "#0277bd",
      "fontSize": "14px"
    }
  }
}%%
graph LR
    subgraph "Core Entities"
        POL[Politician<br/>349 active<br/>~9KB each]
        PARTY[Party<br/>8 parliamentary<br/>~7KB each]
        COMM[Committee<br/>15 riksdag<br/>~17KB each]
        MIN[Ministry<br/>12 government<br/>~19KB each]
    end
    
    subgraph "Intelligence Products"
        RISK[Risk Assessment<br/>50 behavioral rules<br/>~14KB]
        COAL[Coalition Alignment<br/>Party alignment matrix<br/>~9KB]
        TREND[Temporal Trends<br/>Multi-timeframe<br/>~13KB]
    end
    
    subgraph "Advanced Analysis"
        CAREER[Career Path<br/>10-level hierarchy<br/>~5KB]
        ELECTION[Election Proximity<br/>5-phase tracking<br/>~11KB]
        LONGIT[Longitudinal<br/>Multi-semester<br/>~10KB]
    end
    
    subgraph "Relationships"
        REL_POL_PARTY[Member of]
        REL_POL_COMM[Serves on]
        REL_POL_MIN[Minister in]
        REL_PARTY_COAL[Aligns with]
        REL_COMM_MIN[Oversees]
    end
    
    POL -->|REL_POL_PARTY| PARTY
    POL -->|REL_POL_COMM| COMM
    POL -->|REL_POL_MIN| MIN
    PARTY -->|REL_PARTY_COAL| PARTY
    COMM -->|REL_COMM_MIN| MIN
    
    POL -.->|Generates| RISK
    PARTY -.->|Generates| COAL
    POL -.->|Generates| TREND
    
    POL -.->|Tracked by| CAREER
    POL -.->|Analyzed by| ELECTION
    PARTY -.->|Tracked by| LONGIT
    
    style POL fill:#bbdefb,stroke:#1976d2,stroke-width:2px
    style PARTY fill:#c8e6c9,stroke:#388e3c,stroke-width:2px
    style COMM fill:#ffccbc,stroke:#d84315,stroke-width:2px
    style MIN fill:#fff9c4,stroke:#f57f17,stroke-width:2px
    style RISK fill:#f8bbd0,stroke:#c2185b,stroke-width:2px
    style CAREER fill:#e1bee7,stroke:#7b1fa2,stroke-width:2px

Entity Relationships:

  • Politicians are members of parties and serve on committees
  • Some politicians are ministers in government ministries
  • Committees oversee specific ministries
  • Parties form coalitions and align on voting
  • All entities generate intelligence products and analytics

Schema Versions and Field Validation

The JSON export system maintains two versions of each schema to handle the difference between planned features and current implementation:

Original Schemas (Projected Features)

  • Files: *-schema.md (e.g., politician-schema.md)
  • Purpose: Define the complete vision for JSON exports including future features
  • Fields: Include both implemented and planned analytical fields
  • Usage: Guide for API development and feature roadmap

Data-Validated Schemas (Current Reality)

  • Files: *-schema-validated.md (e.g., politician-schema-validated.md)
  • Purpose: Reflect actual available data in current database
  • Fields: Only fields with data in sample CSV files (138 files analyzed)
  • Usage: Accurate representation of what can be exported today
  • Summary: VALIDATED_SCHEMAS_SUMMARY.md

Field Mismatch Summary (Original vs. Validated):

  • Politician: 46 projected → 12 validated (74% not yet implemented)
  • Party: 43 projected → 3 validated (93% not yet implemented)
  • Committee: 26 projected → 2 validated (92% not yet implemented)
  • Ministry: 29 projected → 1 validated (97% not yet implemented)
  • Intelligence: Fully aligned ✅ (0 mismatches)

Why mismatches exist: The original schemas were created as a forward-looking specification before all database views were populated with data. The validated schemas show what's currently available, while original schemas guide future implementation.

Detailed Schema Documentation

All schemas are documented in Markdown format with:

  • Entity structure and field definitions
  • Database view mappings (from 85 available views)
  • Sample data with real examples
  • Mermaid diagrams for visual representation
  • Intelligence applications and use cases

Complete Schema Catalog:

  1. schemas/politician-schema.md - 19,893 bytes, comprehensive politician data model
  2. schemas/party-schema.md - 19,748 bytes, party performance and coalition analytics
  3. schemas/committee-schema.md - 10,245 bytes, committee composition and proposals
  4. schemas/ministry-schema.md - 10,949 bytes, government ministry performance
  5. schemas/intelligence-schema.md - 17,122 bytes, risk assessment and predictive models

Database View Integration: All schemas map to specific database views documented in DATABASE_VIEW_INTELLIGENCE_CATALOG.md (85 views total).

Example JSON Files (CDN-Ready)

The following example files demonstrate the complete JSON export format for each intelligence product:

Core Entity Examples

Example FileSchemaDescriptionFile SizeUpdate Frequency
politician-example.jsonpolitician-schema.mdComprehensive politician profiles with voting, activity, and risk data~9 KBDaily
party-example.jsonparty-schema.mdParty performance, coalition, and electoral analytics~7 KBDaily
committee-example.jsoncommittee-schema.mdCommittee composition, productivity, and decision patterns~17 KBDaily
ministry-example.jsonministry-schema.mdMinistry organization, budget, and performance metrics~19 KBDaily

Intelligence Product Examples

Example FileSchemaDescriptionFile SizeUpdate Frequency
risk-assessment-example.jsonintelligence-schema.mdRisk assessment with 50 behavioral rules catalog~14 KBDaily
coalition-alignment-example.jsonintelligence-schema.mdCoalition alignment matrix and party voting patterns~9 KBDaily
temporal-trends-example.jsonintelligence-schema.mdTemporal trend analysis (daily/weekly/monthly/annual)~13 KBDaily

Advanced Analysis Examples (v1.58-v1.61)

Example FileSchemaDescriptionFile SizeUpdate Frequency
politician-career-path-example.jsonpolitician-schema.mdCareer progression with 10-level hierarchy and risk assessment~5 KBDaily
election-proximity-trends-example.jsonintelligence-schema.mdQuarterly activity tracking relative to election dates~11 KBDaily
party-longitudinal-performance-example.jsonparty-schema.mdMulti-semester performance trends with forecasting~10 KBDaily

📋 Complete Documentation: See examples/EXAMPLES_README.md for comprehensive guide to all examples.

Intelligence Products Coverage:

  • ✅ Risk Assessments: Complete 50-rule risk catalog with entity-level assessments
  • ✅ Coalition Analysis: Party alignment matrix with voting cohesion metrics
  • ✅ Temporal Trends: Multi-timeframe trend analysis for intelligence monitoring
  • ✅ Politician Profiles: Individual politician data with risk and performance scores
  • ✅ Party Analytics: Party-level performance and coalition stability metrics
  • ✅ Committee Analysis: Productivity, decision patterns, and influence metrics
  • ✅ Ministry Performance: Budget execution, policy effectiveness, and predictions
  • ✅ Career Progression: 10-level career path with behavioral and network metrics
  • ✅ Election Cycle Analysis: Pre-election activity surge detection and phase classification
  • ✅ Longitudinal Trends: Multi-semester performance tracking with forecasting indicators

✅ Field Completeness Validation

Overview

The JSON export system includes automated validation to ensure sample CSV files contain all required fields for JSON transformation. This prevents API failures and ensures data product completeness.

Validation Script

Location: json-export-specs/validate-field-completeness.sh

Purpose: Validates that sample CSV files contain the required database columns to support all 5 JSON export schemas.

Running Validation

# From json-export-specs directory
bash validate-field-completeness.sh

# With detailed output
bash validate-field-completeness.sh --verbose

# Generate markdown report (automatic)
# Report saved to: FIELD_COMPLETENESS_REPORT.md

Validation Results

SchemaPrimary CSVRequired FieldsStatus
politician-schema.mdview_riksdagen_politician_sample.csv21 fields✅ 100%
party-schema.mdview_riksdagen_party_sample.csv6 fields✅ 100%
committee-schema.mdview_riksdagen_committee_sample.csv7 fields✅ 100%
ministry-schema.mdview_riksdagen_goverment_sample.csv8 fields✅ 100%
intelligence-schema.mdtable_rule_violation_sample.csv9 fields✅ 100%

Overall Completeness: 100% (all required fields present across all schemas)

Schema-to-CSV Mappings

Each JSON schema maps to specific database views and tables:

Politician Schema

  • Primary CSV: view_riksdagen_politician_sample.csv (sample extract; row counts may vary, column structure validated by CI)
  • Required Fields: person_id, first_name, last_name, party, born_year, gender, total_days_served, active, and 13 more
  • Additional CSVs: view_riksdagen_vote_data_ballot_politician_summary_sample.csv, view_riksdagen_politician_document_summary_sample.csv

Party Schema

  • Primary CSV: view_riksdagen_party_sample.csv (sample extract; row counts may vary, column structure validated by CI)
  • Required Fields: party_id, party_name, head_count, total_documents, documents_last_year, registered_date
  • Additional CSVs: view_riksdagen_party_summary_sample.csv, view_riksdagen_party_ballot_support_annual_summary_sample.csv

Committee Schema

  • Primary CSV: view_riksdagen_committee_sample.csv (sample extract; row counts may vary, column structure validated by CI)
  • Required Fields: embedded_id_org_code, current_member_size, active, total_documents, total_assignments, first/last_assignment_date
  • Additional CSVs: view_committee_productivity_sample.csv

Ministry Schema

  • Primary CSV: view_riksdagen_goverment_sample.csv (sample extract; row counts may vary, column structure validated by CI)
  • Required Fields: name_id, active, current_member_size, total_documents, total_propositions, total_government_bills, first/last_assignment_date
  • Additional CSVs: view_ministry_decision_impact_sample.csv

Intelligence Schema

  • Primary CSV: table_rule_violation_sample.csv (sample extract; row counts may vary, column structure validated by CI)
  • Required Fields: rule_name, reference_id, resource_type, status, name, rule_description, rule_group, detected_date, positive
  • Additional CSVs: view_riksdagen_politician_summary_sample.csv, view_riksdagen_party_summary_sample.csv, and decision flow views

CI Integration

Field completeness validation runs automatically on:

  • Pull requests modifying JSON schemas or sample CSV files
  • Pushes to master/develop branches
  • Changes to the validation script itself

Workflow: .github/workflows/validate-field-completeness.yml

Uses bash script validation following repository patterns with harden-runner security.

Troubleshooting

Missing Fields:

  1. Check database view definition for correct column names
  2. Verify sample CSV extraction query includes all required fields
  3. Re-extract sample data: SELECT * FROM view_name LIMIT 50
  4. Re-run validation to confirm completeness: bash validate-field-completeness.sh

CSV Not Found:

  1. Verify view exists in database schema
  2. Check sample-data directory for correct filename
  3. Extract sample if missing: psql -d cia_dev -c "COPY (SELECT * FROM view_name LIMIT 50) TO '/path/sample.csv' CSV HEADER"
  4. Re-run validation: bash validate-field-completeness.sh

Field Name Mismatch:

  1. Database columns use snake_case (e.g., first_name)
  2. JSON schemas use camelCase (e.g., firstName)
  3. Validation checks database column names, not JSON field names

🏷️ Data Categories & Labels

Label System Architecture

%%{
  init: {
    "theme": "base",
    "themeVariables": {
      "primaryColor": "#e3f2fd",
      "primaryTextColor": "#1565c0",
      "lineColor": "#42a5f5",
      "secondaryColor": "#f3e5f5",
      "tertiaryColor": "#fff3e0",
      "primaryBorderColor": "#1565c0",
      "fontSize": "14px"
    }
  }
}%%
graph TB
    ROOT[Entity Root]
    
    subgraph "🏛️ Politicians"
        POL_CAT[Category: Politician]
        POL_SUB1[Active Member]
        POL_SUB2[Former Member]
        POL_SUB3[Minister]
        POL_SUB4[Committee Chair]
    end
    
    subgraph "🎭 Parties"
        PARTY_CAT[Category: Party]
        PARTY_SUB1[Government Coalition]
        PARTY_SUB2[Opposition]
        PARTY_SUB3[Left Wing]
        PARTY_SUB4[Right Wing]
        PARTY_SUB5[Center]
    end
    
    subgraph "🏢 Ministries"
        MIN_CAT[Category: Ministry]
        MIN_SUB1[Policy Area]
        MIN_SUB2[Budget Size]
        MIN_SUB3[Coalition Member]
    end
    
    subgraph "👥 Committees"
        COM_CAT[Category: Committee]
        COM_SUB1[Standing Committee]
        COM_SUB2[Policy Domain]
        COM_SUB3[Activity Level]
    end
    
    ROOT --> POL_CAT
    ROOT --> PARTY_CAT
    ROOT --> MIN_CAT
    ROOT --> COM_CAT
    
    POL_CAT --> POL_SUB1
    POL_CAT --> POL_SUB2
    POL_CAT --> POL_SUB3
    POL_CAT --> POL_SUB4
    
    PARTY_CAT --> PARTY_SUB1
    PARTY_CAT --> PARTY_SUB2
    PARTY_CAT --> PARTY_SUB3
    PARTY_CAT --> PARTY_SUB4
    PARTY_CAT --> PARTY_SUB5
    
    MIN_CAT --> MIN_SUB1
    MIN_CAT --> MIN_SUB2
    MIN_CAT --> MIN_SUB3
    
    COM_CAT --> COM_SUB1
    COM_CAT --> COM_SUB2
    COM_CAT --> COM_SUB3
    
    style POL_CAT fill:#bbdefb,stroke:#1976d2,stroke-width:3px
    style PARTY_CAT fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
    style MIN_CAT fill:#fff9c4,stroke:#f57f17,stroke-width:3px
    style COM_CAT fill:#ffccbc,stroke:#d84315,stroke-width:3px

Category Definitions

Politicians

  • Primary Categories: Active Member, Former Member, Minister, Committee Chair, Party Leader
  • Subcategories: Electoral District, Party Affiliation, Committee Memberships, Cabinet Positions
  • Intelligence Tags: High Risk, Rising Star, Coalition Broker, Policy Expert, Media Influencer

Parties

  • Primary Categories: Government Coalition, Opposition, Parliamentary, Extra-Parliamentary
  • Subcategories: Ideological Spectrum (Left, Center, Right), Policy Focus, Coalition History
  • Intelligence Tags: Stable, Volatile, Growing, Declining, Swing Voter Appeal

Ministries

  • Primary Categories: Cabinet Ministry, Administrative Agency, Policy Domain
  • Subcategories: Budget Allocation, Staff Size, Coalition Control, Reform Agenda
  • Intelligence Tags: High Performance, Budget Concerns, Reform Focus, Crisis Management

Committees

  • Primary Categories: Standing Committee, Special Committee, EU Committee
  • Subcategories: Policy Domain, Activity Level, Cross-Party Cooperation
  • Intelligence Tags: Productive, Contentious, Consensus-Building, Media Focus

📁 File Structure & Naming

Directory Layout

json-export/
├── v1.0.0/                          # Version directory
│   ├── metadata.json                # Export metadata
│   ├── politicians/
│   │   ├── index.json              # All politicians list
│   │   ├── active.json             # Active politicians only
│   │   ├── by-party/
│   │   │   ├── s.json              # Social Democrats
│   │   │   ├── m.json              # Moderates
│   │   │   └── ...
│   │   ├── by-committee/
│   │   │   ├── au.json             # Foreign Affairs
│   │   │   ├── ju.json             # Justice
│   │   │   └── ...
│   │   └── profiles/
│   │       ├── {personId}.json     # Individual profiles
│   │       └── ...
│   ├── parties/
│   │   ├── index.json              # All parties
│   │   ├── parliamentary.json      # Parliamentary parties
│   │   ├── profiles/
│   │   │   ├── {partyId}.json      # Individual party profiles
│   │   │   └── ...
│   │   └── analytics/
│   │       ├── voting-cohesion.json
│   │       ├── electoral-trends.json
│   │       └── coalition-matrix.json
│   ├── ministries/
│   │   ├── index.json
│   │   ├── current-government.json
│   │   ├── profiles/
│   │   │   ├── {ministryId}.json
│   │   │   └── ...
│   │   └── performance/
│   │       ├── budget-execution.json
│   │       └── policy-effectiveness.json
│   ├── committees/
│   │   ├── index.json
│   │   ├── standing.json
│   │   ├── profiles/
│   │   │   ├── {committeeId}.json
│   │   │   └── ...
│   │   └── productivity/
│   │       ├── legislation-output.json
│   │       └── decision-timeline.json
│   └── intelligence/
│       ├── risk-assessments.json
│       ├── trend-analysis.json
│       ├── coalition-stability.json
│       ├── voting-patterns.json
│       └── predictive-analytics.json
└── latest/                          # Symlink to current version
    └── ... (mirrors v1.0.0 structure)

Naming Conventions

  • Version Format: Semantic versioning (MAJOR.MINOR.PATCH)
  • File Names: Lowercase, hyphen-separated, descriptive
  • IDs: Use official identifiers from source systems
  • Timestamps: ISO 8601 format (UTC)

🌐 CDN Deployment Guide

Cost: ~$15-30/month for 100GB storage + 1TB transfer

# 1. Create S3 bucket
aws s3 mb s3://cia-json-export --region eu-north-1

# 2. Configure public read access
aws s3api put-bucket-policy --bucket cia-json-export --policy '{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "PublicReadGetObject",
    "Effect": "Allow",
    "Principal": "*",
    "Action": "s3:GetObject",
    "Resource": "arn:aws:s3:::cia-json-export/*"
  }]
}'

# 3. Enable CORS
aws s3api put-bucket-cors --bucket cia-json-export --cors-configuration '{
  "CORSRules": [{
    "AllowedOrigins": ["*"],
    "AllowedMethods": ["GET", "HEAD"],
    "AllowedHeaders": ["*"],
    "MaxAgeSeconds": 3600
  }]
}'

# 4. Create CloudFront distribution
aws cloudfront create-distribution --origin-domain-name cia-json-export.s3.amazonaws.com

Configuration:

  • Cache Behavior: 24-hour TTL
  • Compression: Gzip enabled
  • HTTPS: Required
  • Query String Forwarding: None
  • Geo Restrictions: None

Option 2: Cloudflare Pages

Cost: Free tier (unlimited bandwidth)

  1. Create Cloudflare account
  2. Add custom domain or use pages.dev subdomain
  3. Deploy via Git integration or Wrangler CLI
  4. Configure caching rules and CORS headers

Option 3: Netlify

Cost: Free tier (100GB bandwidth/month)

# Deploy with Netlify CLI
npm install -g netlify-cli
netlify deploy --prod --dir=json-export

CORS Configuration

{
  "Access-Control-Allow-Origin": "*",
  "Access-Control-Allow-Methods": "GET, HEAD, OPTIONS",
  "Access-Control-Allow-Headers": "Content-Type",
  "Access-Control-Max-Age": "86400",
  "Cache-Control": "public, max-age=86400"
}

💻 JavaScript Usage Examples

Basic Fetch Example

// Fetch all politicians
async function fetchPoliticians() {
  const response = await fetch('https://cdn.cia.se/v1.0.0/politicians/index.json');
  const data = await response.json();
  
  console.log(`Loaded ${data.metadata.recordCount} politicians`);
  return data.data;
}

// Fetch specific politician profile
async function fetchPoliticianProfile(personId) {
  const response = await fetch(`https://cdn.cia.se/v1.0.0/politicians/profiles/${personId}.json`);
  return response.json();
}

React Integration

import { useState, useEffect } from 'react';

function PoliticianList() {
  const [politicians, setPoliticians] = useState([]);
  const [loading, setLoading] = useState(true);
  
  useEffect(() => {
    fetch('https://cdn.cia.se/v1.0.0/politicians/active.json')
      .then(res => res.json())
      .then(data => {
        setPoliticians(data.data);
        setLoading(false);
      })
      .catch(err => console.error('Failed to load politicians:', err));
  }, []);
  
  if (loading) return <div>Loading...</div>;
  
  return (
    <div>
      <h2>Active Politicians ({politicians.length})</h2>
      <ul>
        {politicians.map(pol => (
          <li key={pol.id}>
            {pol.attributes.firstName} {pol.attributes.lastName} 
            ({pol.attributes.party})
            <p>{pol.labels.descriptions.short}</p>
          </li>
        ))}
      </ul>
    </div>
  );
}

Vue.js Integration

<template>
  <div>
    <h2>Parliamentary Parties</h2>
    <div v-for="party in parties" :key="party.id">
      <h3>{{ party.attributes.name }}</h3>
      <p>{{ party.labels.descriptions.long }}</p>
      <span>Members: {{ party.intelligence.memberCount }}</span>
    </div>
  </div>
</template>

<script>
export default {
  data() {
    return {
      parties: []
    };
  },
  async mounted() {
    const response = await fetch('https://cdn.cia.se/v1.0.0/parties/parliamentary.json');
    const data = await response.json();
    this.parties = data.data;
  }
};
</script>

Data Visualization (D3.js)

// Load and visualize party voting cohesion
async function visualizePartyVoting() {
  const response = await fetch('https://cdn.cia.se/v1.0.0/parties/analytics/voting-cohesion.json');
  const data = await response.json();
  
  const svg = d3.select('#chart')
    .append('svg')
    .attr('width', 800)
    .attr('height', 400);
  
  // Create bar chart showing party cohesion scores
  svg.selectAll('rect')
    .data(data.data)
    .enter()
    .append('rect')
    .attr('x', (d, i) => i * 90 + 50)
    .attr('y', d => 350 - d.intelligence.cohesionScore * 3)
    .attr('width', 70)
    .attr('height', d => d.intelligence.cohesionScore * 3)
    .attr('fill', d => d.attributes.color);
}

🔄 Update & Versioning Strategy

Update Schedule

%%{
  init: {
    "theme": "base",
    "themeVariables": {
      "primaryColor": "#fff9c4",
      "primaryTextColor": "#f57f17",
      "lineColor": "#fbc02d",
      "secondaryColor": "#e1bee7",
      "tertiaryColor": "#e8f5e9",
      "primaryBorderColor": "#f57f17",
      "fontSize": "14px"
    }
  }
}%%
gantt
    title Daily Update Cycle
    dateFormat HH:mm
    axisFormat %H:%M
    
    section Data Collection
    Extract from DB Views    :a1, 01:00, 30m
    Transform to JSON        :a2, after a1, 20m
    Validate Schema          :a3, after a2, 10m
    
    section Versioning
    Generate Version Tag     :b1, after a3, 5m
    Create Archive           :b2, after b1, 10m
    
    section Deployment
    Upload to S3             :c1, after b2, 15m
    Invalidate CDN Cache     :c2, after c1, 5m
    Update Latest Symlink    :c3, after c2, 2m
    
    section Validation
    Health Check             :d1, after c3, 10m
    Send Notifications       :d2, after d1, 2m

Semantic Versioning

  • MAJOR: Breaking schema changes (incompatible with previous versions)
  • MINOR: New fields added, backward compatible
  • PATCH: Bug fixes, data corrections, no schema change

Version Migration

// Version compatibility checker
function checkVersion(requiredVersion, currentVersion) {
  const [reqMajor] = requiredVersion.split('.');
  const [curMajor] = currentVersion.split('.');
  return reqMajor === curMajor;
}

// Fetch with version fallback
async function fetchWithFallback(path, preferredVersion = 'latest') {
  try {
    const url = `https://cdn.cia.se/${preferredVersion}/${path}`;
    return await fetch(url).then(r => r.json());
  } catch (err) {
    console.warn(`Failed to fetch ${preferredVersion}, trying v1.0.0`);
    return await fetch(`https://cdn.cia.se/v1.0.0/${path}`).then(r => r.json());
  }
}

🔗 API Compatibility

RESTful URL Patterns

The JSON export structure mimics RESTful API conventions:

PatternExampleDescription
/{entity}/index.json/politicians/index.jsonList all entities
/{entity}/{filter}.json/politicians/active.jsonFiltered list
/{entity}/by-{field}/{value}.json/politicians/by-party/s.jsonGroup by field
/{entity}/profiles/{id}.json/politicians/profiles/0123456789.jsonIndividual profile
/{entity}/{category}/{metric}.json/parties/analytics/voting-cohesion.jsonAnalytical data

Query Parameters (Future Enhancement)

While static JSON doesn't support query parameters, future enhancements could include:

  • Pre-generated filtered views for common queries
  • GraphQL layer for complex queries
  • API Gateway with caching tier

📊 Performance Optimization

File Size Optimization

OptimizationStrategySavings
MinificationRemove whitespace~15-20%
Gzip CompressionCDN-level compression~70-80%
PaginationSplit large datasetsVariable
Selective FieldsMinimal vs. full profiles~40-50%
Lazy LoadingLoad on-demandN/A

Caching Strategy

// Service Worker caching for offline support
self.addEventListener('fetch', event => {
  if (event.request.url.includes('cdn.cia.se')) {
    event.respondWith(
      caches.open('cia-data-v1').then(cache => {
        return cache.match(event.request).then(response => {
          return response || fetch(event.request).then(response => {
            cache.put(event.request, response.clone());
            return response;
          });
        });
      })
    );
  }
});

🔐 Security Considerations

Data Privacy

  • Only public information included (no personal contact details)
  • Compliance with GDPR and Swedish data protection laws
  • No tracking or analytics in JSON files

Access Control

  • Public read access (no authentication required)
  • HTTPS-only (TLS 1.2+ required)
  • Rate limiting at CDN level
  • No write access from client side

Integrity Verification

// Verify data integrity with checksum
async function verifyDataIntegrity(url, expectedHash) {
  const response = await fetch(url);
  const data = await response.text();
  
  const buffer = new TextEncoder().encode(data);
  const hashBuffer = await crypto.subtle.digest('SHA-256', buffer);
  const hashArray = Array.from(new Uint8Array(hashBuffer));
  const hashHex = hashArray.map(b => b.toString(16).padStart(2, '0')).join('');
  
  return hashHex === expectedHash;
}

📚 Additional Resources

Schema Documentation

Tools & Libraries

🤝 Contributing

For questions, issues, or contributions related to the JSON export system:

  1. Review existing schemas and documentation
  2. Test changes with sample data
  3. Validate JSON schema compliance
  4. Submit pull request with clear description
  5. Ensure backward compatibility

📝 License

The JSON export system and generated data files are licensed under the Apache License 2.0, consistent with the Citizen Intelligence Agency platform.

Last Updated: 2024-11-24
Maintained By: Citizen Intelligence Agency Development Team
Contact: See CONTRIBUTING.md