🪞 Methodology Reflection Template

May 5, 2026 · View on GitHub

📌 Template Instructions: Copy to analysis/daily/{date}/{article-type}-run{N}/intelligence/methodology-reflection.md. Produced as the final artifact of every run — after workflow-audit.md and after the PR patch is finalised. See methodologies/per-artifact-methodologies.md §methodology-reflection. Modelled on the riksdagsmonitor interpellations reflection.

🎯 Purpose: Retrospective on the analytic pipeline itself — what worked, what did not, which Structured Analytic Techniques (SATs) were applied, where biases were mitigated, and what the next same-type run must do differently. Distinct from workflow-audit (mechanical phase-by-phase compliance) — this file is the analytic-quality reflection that closes the learning loop.

📋 Document Metadata

Field	Value
Analysis Date	`[REQUIRED: YYYY-MM-DD]`
Workflow	`[REQUIRED: e.g. news-breaking (agentic workflow) + reference-class expansion]`
Article Type	`[REQUIRED]`
Run ID	`[REQUIRED]`
AI-FIRST iterations	`[REQUIRED: minimum 2 — Pass 1 + Pass 2; note any Pass 3 expansion]`
Confidence (overall)	`[REQUIRED: 🟢 HIGH / 🟡 MEDIUM / 🔴 LOW]`

1️⃣ Pipeline Overview

%%{init: {"theme":"dark","themeVariables":{"primaryColor":"#1565C0","primaryTextColor":"#ffffff","primaryBorderColor":"#0A3F7F","lineColor":"#90CAF9","secondaryColor":"#2E7D32","secondaryTextColor":"#ffffff","tertiaryColor":"#FF9800","tertiaryTextColor":"#000000","mainBkg":"#1565C0","secondBkg":"#2E7D32","tertiaryBkg":"#FF9800","noteBkgColor":"#FFC107","noteTextColor":"#000000","errorBkgColor":"#D32F2F","fontFamily":"Inter, Helvetica, Arial, sans-serif"}}}%%
graph TD
    A[Trigger: scheduled agentic workflow] --> B[MCP data pull: ep-mcp / world-bank / imf]
    B --> C[Phase 1 data collection]
    C --> D{Health gate: MCP reachable?}
    D -->|yes| E[Phase 2 editorial context]
    D -->|degraded| F[Degraded-mode bridge: WB/IMF + prior-run cache]
    E --> G[Phase 3 analysis — classification + SWOT + risk + threat + stakeholder]
    F --> G
    G --> H[AI-FIRST Pass 1]
    H --> I[Self-review: depth, evidence, confidence grading]
    I --> J[AI-FIRST Pass 2: improvement]
    J --> K[Phase 4 significance gate]
    K --> L[Phase 5 validation]
    L --> M[Phase 6 PR creation]
    M --> N[Human editorial review]
    N -->|feedback| O[Pass 3 expansion if required]
    O --> P[Final publish]

    style A fill:#1565C0,color:#ffffff
    style D fill:#FFC107,color:#000000
    style F fill:#FF9800,color:#000000
    style H fill:#7B1FA2,color:#ffffff
    style J fill:#7B1FA2,color:#ffffff
    style K fill:#FF9800,color:#000000
    style L fill:#2E7D32,color:#ffffff
    style M fill:#2E7D32,color:#ffffff
    style N fill:#0288D1,color:#ffffff
    style P fill:#2E7D32,color:#ffffff

[REQUIRED: customise this diagram with the run's actual data sources (ep-mcp tools used, world-bank / imf fallbacks, any direct REST bridges), actual iteration count, and editorial-review outcome.]

1️⃣.5 Data Mode Declaration

Field	Value
`dataMode`	`[REQUIRED: full / title-only / degraded-imf / degraded-voting / minimal]`
Justification	`[REQUIRED: why this mode — which probes failed, which data is structurally unavailable]`
Line-floor reduction applied	`[REQUIRED: 0% / 15% / 15% / 25% / 35% — for full / degraded-imf / degraded-voting / title-only / minimal per reference-quality-thresholds.json §1.4.0]`
Confidence ceiling	`[REQUIRED: when non-full, state the maximum confidence level permitted for data-dependent claims]`

Stage B Scaffold Checklist — MANDATORY at minute 0 of Stage B:

Before writing any analytical content, create empty stub files for every mandatory artifact. This ensures Pass 1 produces all files (even if below floor) and Pass 2 can deepen rather than create from scratch.

Scaffold protocol:

Read the mandatory artifact list from artifact-catalog.md for the current articleType
For every mandatory artifact: create ${ANALYSIS_DIR}/${relativePath} containing the SPDX header, an H1 title matching the template, and a  comment for intelligence/, classification/, risk-scoring/, and threat-assessment/ files
Record scaffold timestamp in manifest: "scaffoldedAt": "<ISO timestamp>"
Only THEN begin analytical content generation in Pass 1

[REQUIRED: confirm scaffold was performed — timestamp and artifact count. If NOT performed, explain why and document the time cost of creating files in Pass 2 instead.]

2️⃣ Data Sources and Provenance

Source	Purpose	Status	Confidence grade
`ep-mcp` — `[tool]`	`[REQUIRED]`	`[✅ Worked / ⚠️ Degraded / ❌ Failed]`	`[🟢/🟡/🔴]`
`ep-mcp` — `[tool]`	`[REQUIRED]`	`[...]`	`[...]`
`ep-mcp` — `[tool]`	`[REQUIRED]`	`[...]`	`[...]`
`world-bank-mcp` — `[indicator]`	Macro context	`[✅/⚠️/❌]`	`[🟢/🟡/🔴]`
`imf-native` (REST) — `[series]`	`[REQUIRED: primary economic source under + per-type indicator-floor satisfaction per imf-indicator-mapping.md §8]`	`[✅/⚠️/❌]`	`[🟢/🟡/🔴]`
External (e.g. EUR-Lex, Commission press)	`[REQUIRED]`	`[✅/⚠️/❌]`	`[🟧 MEDIUM if external]`

≥6 rows required. Always note when a feed returned HTML instead of JSON, or returned 0 results where a non-zero response was expected (0 results can itself be high-signal intelligence data).

3️⃣ Structured Analytic Techniques Applied

Technique	Artifact(s)	Value delivered
Classification (7-dimension)	`classification/significance-classification.md`	`[REQUIRED: ≥10 words]`
Significance scoring (5-dimension composite)	`intelligence/significance-scoring.md`	`[REQUIRED]`
SWOT (quantitative, TOWS)	`risk-scoring/quantitative-swot.md`	`[REQUIRED]$
\text{Risk} \text{matrix} (\text{L} \times \text{I}, 1–5)	$risk-scoring/risk-matrix.md`	`[REQUIRED]`
Threat analysis (multi-framework)	`intelligence/threat-model.md`	`[REQUIRED]`
Stakeholder mapping (power × alignment)	`intelligence/stakeholder-map.md`	`[REQUIRED]`
PESTLE	`intelligence/pestle-analysis.md`	`[REQUIRED]`
Scenario analysis (probability-weighted)	`intelligence/scenario-forecast.md`	`[REQUIRED]`
ACH — Analysis of Competing Hypotheses	`[REQUIRED: artifact or section name]`	`[REQUIRED]`
Key Assumptions Check	`[REQUIRED]`	`[REQUIRED]`
Red Team / Devil's Advocate	`[REQUIRED]`	`[REQUIRED]`
Historical baseline (30/90-day)	`intelligence/historical-baseline.md`	`[REQUIRED]`
Cross-run Bayesian delta	`intelligence/cross-run-diff.md`	`[REQUIRED]`
Comparative international (EU-27 peer)	`[REQUIRED]`	`[REQUIRED]`
Per-document deep dives	`documents/document-analysis-index.md` + per-file `.analysis.md`	`[REQUIRED]`

Ensure every SAT is either marked applied with its artifact, or explicitly noted as "not-applicable this run because …".

4️⃣ AI-FIRST Iteration Log

The AI-FIRST principle mandates minimum 2 complete iterations with genuine critical re-evaluation between iterations.

Pass 1 — Initial generation (~`[REQUIRED: minutes]` of allocated compute)

[REQUIRED: count + list of artifacts produced]
[REQUIRED: coverage status per artifact group]
[REQUIRED: Mermaid diagrams included — basic / themed]

Self-evaluation of Pass 1:

Coverage gaps: [REQUIRED]
Depth gaps: [REQUIRED: which artifacts fell short of depth floor]
SATs missing: [REQUIRED]
Evidence density: [REQUIRED]
Confidence grading completeness: [REQUIRED]

Pass 2 — Improvement iteration (~`[REQUIRED: minutes]`)

[REQUIRED: concrete changes made — named sections expanded, new citations added, Mermaid upgraded]
[REQUIRED: specific word / citation / RCV-ID counts added]

Gaps identified during Pass 2 (carry-forward or deferred to Pass 3):

[REQUIRED: list or state "no further gaps"]

Pass 3 — Reference-class expansion (only if editorial review triggered it)

[REQUIRED: document trigger — reviewer feedback reference, significance-gate miss, or "not triggered this run"]

Actions taken:

[REQUIRED]
[REQUIRED]
[REQUIRED]

5️⃣ Strengths of This Analysis

[REQUIRED: ≥30 words — name the strongest evidence class used this run with ≥1 specific citation]
[REQUIRED: ≥30 words — quantitative anchoring; name indicators]
[REQUIRED: ≥30 words — pattern detection / novel finding]
[REQUIRED: ≥30 words — SATs breadth]
[REQUIRED: ≥30 words — comparative benchmarking where present]
[REQUIRED: ≥30 words — confidence grading discipline]

≥5 strengths required. Each strength must cite ≥1 specific artifact or data point.

6️⃣ Limitations and Caveats

[REQUIRED: ≥30 words — MCP endpoint that failed or was degraded + what was inferred vs. observed]
[REQUIRED: ≥30 words — data-freshness limits, roll-call publication delay, etc.]
[REQUIRED: ≥30 words — single-run vs. multi-run inference constraints]
[REQUIRED: ≥30 words — polling / sentiment / non-public data unavailable]
[REQUIRED: ≥30 words — language / cultural biases; mitigation if any]
[REQUIRED: ≥30 words — analyst-side bias risk acknowledged]

≥5 limitations required. Every limitation must be honest and specific — not boilerplate.

7️⃣ Lessons for Future Same-Type Runs

[REQUIRED: ≥30 words — data-collection lesson; e.g. "always pull N prior runs for baseline"]
[REQUIRED: ≥30 words — SAT timing lesson; e.g. "apply ACH from Pass 1 to prevent confirmation-bias narrative"]
[REQUIRED: ≥30 words — comparative-context lesson]
[REQUIRED: ≥30 words — methodology artifact lesson — e.g. what reflection would have caught]
[REQUIRED: ≥30 words — iteration-time budget lesson]

≥5 lessons required. Each lesson must be concrete enough that the NEXT same-type run can execute it without re-deriving the insight.

8️⃣ Known Biases and Mitigations

Bias	Risk	Mitigation applied	Residual
Confirmation bias (favouring dominant hypothesis)	`[H/M/L]`	`[REQUIRED: explicit ACH matrix / Red Team / inconsistency counting]`	`[L/M/H]`
Availability bias (over-weighting widely-cited items)	`[H/M/L]`	`[REQUIRED]`	`[L/M/H]`
Mirror-imaging (assuming EP politics mirror analyst reference frame)	`[H/M/L]`	`[REQUIRED: direct quotation / comparative international / multi-language]`	`[L/M/H]`
Narrative fallacy (constructing coherent story from noise)	`[H/M/L]`	`[REQUIRED: Red Team position / coherence challenge]`	`[L/M/H]`
Recency bias (over-weighting last week)	`[H/M/L]`	`[REQUIRED: 30/90-day baseline, prior-session cross-reference]`	`[L/M/H]`
Selection bias (only published items visible)	`[H/M/L]`	`[REQUIRED: withdrawals, null responses, absent signals captured]`	`[L/M/H]`
Anchoring (over-relying on prior-run posterior)	`[H/M/L]`	`[REQUIRED: Bayesian delta with full prior→posterior chain]`	`[L/M/H]`

≥6 biases required. Every mitigation must reference an artifact or section.

9️⃣ Peer Review / Editorial Oversight

Per Hack23 AI_Policy.md, AI-assisted analysis requires human editorial review before publication. Document the oversight chain for this run:

Generated by [REQUIRED: workflow name] agentic workflow (AI)
Reviewed by [REQUIRED: human reviewer handle OR "pending editorial queue"]
Feedback / revisions: [REQUIRED: summary or "none requested"]
Published HTML articles signed-off for production deployment: [REQUIRED: yes/no/pending]

🔟 Update Plan

Trigger	Artifact(s) to update	Frequency
New EP plenary session in window	`intelligence/cross-session-intelligence.md`, `intelligence/coalition-dynamics.md`	Event-driven
Roll-call data publishes (typical lag: 2–6 weeks)	`intelligence/voting-patterns.md`, `intelligence/coalition-dynamics.md`	Event-driven
IMF (primary economic) / World Bank (non-economic) data vintage update	`intelligence/economic-context.md`	Monthly
Commission proposal / trilogue conclusion	`intelligence/scenario-forecast.md`, `risk-scoring/legislative-velocity-risk.md`	Event-driven
Quarterly reference-class review	All top-level artifacts	Quarterly
`[REQUIRED: run-specific trigger]`	`[REQUIRED]`	`[REQUIRED]`

≥5 triggers required.

1️⃣1️⃣ References

Heuer, R. J. (1999). Psychology of Intelligence Analysis — CIA Center for the Study of Intelligence.
Heuer, R. J., & Pherson, R. H. (2020). Structured Analytic Techniques for Intelligence Analysis (3rd ed.) — CQ Press.
UK MoD Red Teaming Handbook (2021).
NATO Intelligence Handbook (AJP-2.1).
Hack23 AI_Policy.md — ISMS-PUBLIC.
Hack23 Secure Development Policy.
.github/skills/ai-first-quality.md — AI-FIRST principle + 2-pass rule.
analysis/methodologies/ai-driven-analysis-guide.md — 10-step protocol.
[REQUIRED: run-specific references — WB/IMF series IDs, EU Commission docs, EUR-Lex entries]

1️⃣2️⃣ ICD 203 Tradecraft Compliance

Purpose: Verify that every probabilistic claim in this run meets the professional estimative-language and source-grading standards codified in osint-tradecraft-standards.md.

Compliance Dimension	Standard	Status	Evidence / Notes
WEP band usage	Every uncertain judgement uses a WEP band (ICD 203 / Kent §3.1)	`[✅ / ⚠️ / ❌]`	`[REQUIRED: how many headline judgements carry a WEP band?]`
Time horizon	Every WEP band has an explicit time horizon (§3.4)	`[✅ / ⚠️ / ❌]`	`[REQUIRED: any bands without a time horizon?]`
Admiralty source grades	Every external source carries an A1–F6 Admiralty grade (§2)	`[✅ / ⚠️ / ❌]`	`[REQUIRED: note any un-graded sources]`
Confidence vs. probability separation	Confidence-in-evidence (H/M/L) tracked separately from WEP probability (§3.3)	`[✅ / ⚠️ / ❌]`	`[REQUIRED: any conflated markers?]`
ICD 203 BLUF verb usage	"assess" / "judge" used for analytic positions (not "think"/"believe")	`[✅ / ⚠️ / ❌]`	`[REQUIRED: note any non-standard verbs in BLUF sentences]`
SAT application	≥1 named SAT (§4) documented in this run	`[✅ / ⚠️ / ❌]`	`[REQUIRED: list SATs applied — e.g. ACH, Key Assumptions Check, Pre-Mortem]`
Single-source flag	Claims driven by a source ≤C3 with no A–B corroboration are flagged	`[✅ / ⚠️ / N/A]`	`[REQUIRED: note any un-corroborated D-F grade sources driving headline findings]`

ICD 203 overall verdict: [REQUIRED: COMPLIANT / PARTIALLY COMPLIANT / NON-COMPLIANT]
Remediation required for next run: [REQUIRED: list gaps or state "none"]

✅ Quality Gate (self-check before commit)

Document Control: /analysis/daily/{date}/{type}-run{N}/intelligence/methodology-reflection.md · Template v1.1 · Depth floor: per article-type minimum defined in reference-quality-thresholds.json (authoritative) · Produced as the last artifact of each run, after workflow-audit.md.