Scarcity Benchmark 03: Economic Simulation and Ablations

13. S4 — Economic Simulation: Direction Validation and Uncertainty

Engine trained on Kenya 1990–2023 (34 years). Three shocks propagated 5 steps from 2023 state. Validated against directional coherence with IMF/WB documented relationships. Magnitude is not validated — 34 observations insufficient for magnitude precision.

Shock S1: Electricity access +20 pp (50% → 70%)

S1 direction score: 3/3 unambiguous (100%)

Shock S2: Government debt +15 pp GDP (~55% → ~70%)

S2 direction score: 2/3 unambiguous (67%)

Anomaly note: The negative interest rate response contradicts crowding-out theory but is consistent with Kenya's documented financial repression — the CBK used administered rates during fiscal expansions (IMF Art. IV 2019, 2022). This is empirically grounded even if it violates textbook expectation.

Shock S3: Inflation +5 pp (7.7% → 12.7%)

S3 direction score: 5/5 unambiguous (100%)

Overall

Simulation uncertainty

Multi-seed discovery stability (5 seeds, federated KEN+TZA+UGA):

Discovery quality is highly stable across seeds: coefficient of variation = 1.5% on avg_conf. The hypothesis graph that drives simulation is therefore reproducible.

Binomial 95% CI on the 10/11 direction-match (Clopper-Pearson): The 91% direction-match is based on 11 unambiguous relationships. The binomial 95% CI is approximately [59%, 100%]. This wide interval reflects the small sample of testable relationships, not instability in the engine. External replication on a larger set of economic shocks is the right path to a tighter estimate — this benchmark validates the direction of the contribution, not its precision.

Synthetic data simulation (direction match on random data): Running the same shock tests on synthetic random data (5 seeds) produces ~20% direction match (near-chance). This confirms the 91% on real Kenya data is not an artefact of the simulation machinery: random data → random directions, real economic data → coherent directions.

Comparison to no-discovery baseline: Without Scarcity, the PolicySimulator has no hypothesis graph and returns zero propagation for all shocks. The 91% match is a comparison to no model at all, not to a weaker model.

14. Confidence: External Anchoring

Critical fact: Local-only final confidence = 0.205 (below 0.25). Federated final confidence = 0.298 (above 0.25). This is not marginal — it is the difference between zero and non-zero simulation capability.

15. Ablation Studies

A. Sparsity Sweep

At 60% data drop, federated confidence (0.226) exceeds local confidence at 0% drop (0.154). Federation compensates for losing 60% of observations.

B. Federation Size

Concave benefit curve. First peer dominates.

C. Buffer Size (Annual Data)

No effect at 34 annual observations — buffer is never full. See §11 for high-frequency results.

D. Peer Specificity

All pairs: +0.15 to +0.20. No dominant pair. Federation benefit does not depend on geographic or structural similarity.

E. Lifecycle Management Ablation

Lifecycle management is essential: without it, conf=0.121 (below gate, no simulation). Too tight (conf≥0.5) leaves 2/3 countries with zero active hypotheses at N=34.

F. Confidence Gate Sensitivity

At gate=0.25, the top half of the hypothesis pool qualifies (44–50%). Avg confidence of eligible hypotheses: 0.384.

G. Federation Mechanism Ablation

Evidence sharing captures 65% of centralised advantage without requiring data pooling.

H. Peer Count Ablation (Uganda focus)

First peer gives largest gain. Second peer marginal. Concave returns confirmed.

16. What Is Being Learned

Scarcity discovers 15 relationship types across variable pairs, organised into 4 families:

What is confirmed at annual frequency (N≤34):

All 5 final active hypotheses are TEMPORAL type. Examples from Kenya:

Why only TEMPORAL at N=34:

CAUSAL, MEDIATING, SYNERGISTIC, COMPETITIVE types require sustained cross-variable evidence: the Bayesian accumulator needs N >> 34 consistent observations of the multi-variable pattern to cross the 0.25 confidence gate. At annual frequency, 34 years is sufficient to confirm univariate persistence (TEMPORAL) but not directional cross-variable structure (CAUSAL). CAUSAL and MEDIATING types remain TENTATIVE throughout (conf = 0.125–0.25), accumulating evidence but never crossing the promotion threshold.

Implication: The simulation in §13 propagates shocks through TEMPORAL hypotheses (autoregressive persistence) rather than explicit CAUSAL edges. This is observationally coherent but is not an SCM counterfactual graph. Explicitly causal edges require either higher-frequency data (daily: N>>365) or longer time series (N>>50 annual).

17. Error Analysis — Hardest Indicators

Difficulty > 1 means AR1 is worse than predicting the mean. Real interest rate and exports are hardest — structural shocks (2008 GFC, COVID, CBK policy shifts) invalidate AR(1). These are exactly where cross-variable causal structure adds most value.

18. Temporal Instability — Regime Transfer Test

Method: Train AR1 fixed on 1990–2007 (pre-GFC). Evaluate on 2008–2023 (post-GFC + COVID regime). Compare to AR1 rolling (retrained up to each test year). Also track Scarcity discovery quality across the split.

Scarcity discovery quality:

Findings:

1. AR1 regime degradation: Frozen pre-2008 parameters degrade 4.3% on post-2008 data (on smooth synthetic data). On real data with GFC and COVID structural breaks, this gap would be substantially larger. This is the known non-stationarity problem in macroeconomic forecasting: AR(1) parameters fitted on one regime are wrong in another.

2. Scarcity prediction is immune: Scarcity's lag-1 prediction (use last observed value) does not rely on fitted parameters — it is inherently regime-agnostic. There is no AR1 slope to become stale. Prediction MAE does not degrade under regime change.

3. Scarcity discovery continues post-crisis: Discovery confidence grows +66.5% as the engine continues to observe post-2008 data. Relationships discovered before the crisis that persist after it receive additional confirming evidence; those that break are pruned. The engine does not need a manual reset after regime change — it adapts continuously.

4. Note on synthetic data: This test uses synthetic data, which has smooth gradients and no real structural breaks. On real WB data, the GFC and COVID create genuine step-function changes in some indicators. The 4.3% AR1 degradation is therefore a lower bound estimate of the real-data regime transfer cost. Re-running with --live is recommended.

19. Federation Mechanism — Evidence Sharing vs Parameter Averaging

FedAvg each round:
  all nodes fit local AR(1)
  server averages alpha, beta per indicator → replaces local model

Scarcity each period:
  each node streams its new observation row to peers
  each node processes peer rows through its local hypothesis engine
  hypotheses confirmed by multiple peers accumulate confidence faster
  hypotheses contradicted by peers lose confidence and are pruned
  each node's model is never replaced — only its evidence base grows

FedAvg assumes all nodes learn the same function. Scarcity assumes nodes share structural patterns but may differ in magnitudes, lags, and regimes. Evidence sharing lets each node confirm or deny peer patterns without having peer parameters imposed on it.

Communication cost: 34 rounds for annual data, each round transmitting 19 float32 values per peer (~76 bytes per peer per year). Total per node: 76 × 2 peers × 34 years = 5.2 KB — negligible even for constrained edge deployments.

20. Scenario Experiments

Local vs Federated (all 3 countries, full timeline)

Federated: 2.3× higher confidence, tighter hypothesis set (52–53 vs 63 active).

Late Joiner (Uganda joins 10 years after KEN+TZA)

Warm-start: 2.2× higher. Consistent with Ethiopia result (§10).