Scarcity Benchmark 02: Core Results and Data Scarcity

4. Main Results — Prediction Accuracy

Real World Bank data | 20 seeds × 3 countries × rolling folds | lower MAE = better

Ridge-Lag from dry-run benchmark (synthetic data, single seed); other methods on real WB data. Scarcity-Local and Scarcity-Fed produce identical MAE (same lag-1 mechanism). Federation benefit is in discovery quality, not point prediction.

Finding (C2, C3): FedAvg-AR1 is 28% worse than Local-AR1 despite 3× more training data — parameter averaging across heterogeneous AR(1) slopes degrades both countries' models. Scarcity achieves the best MAE (0.493), beating Oracle-AR1 (0.562). Lag-1 is more robust to structural breaks than fitted AR(1) at N<25.

§4b — The Oracle-Loss Argument (why Scarcity beating Oracle matters)

Oracle-AR1 is the theoretical upper bound of the entire AR(1) model family. It trains on pooled data from all 3 countries (3× the local observations), uses the same rolling fold protocol, and is not achievable without data centralisation (a privacy violation in federated settings).

Scarcity (MAE=0.493) beats Oracle-AR1 (MAE=0.562) by 12.3%.

This is counterintuitive and requires explanation. The prediction mechanism for Scarcity is lag-1 (predict last observed value), whereas AR(1) fits a slope parameter β. At N<25: - Fitted β̂ has high estimation variance — the slope "chases" noise in 5–24 training points - Lag-1 (β≡1) is the correct prediction for nearly random-walk processes at this horizon - Oracle-AR1's pooled data gives a more stable β̂, but the fitted slope still misses structural breaks that lag-1 naturally handles (last value is always correct at t−1)

Scarcity does not beat Oracle because it is a better predictor. Lag-1 is a better predictor of annual macroeconomic series at N<25. This is a known property of random-walk-adjacent processes (Diebold & Mariano 1995). The result is reported honestly: Scarcity's primary contribution is discovery, not prediction accuracy.

5. Discovery Quality

Critical threshold: The 0.25 gate allows get_candidate_paths() to emit hypotheses to the PolicySimulator. Local-only confidence (0.205) never crosses this threshold. Federation is not an enhancement — it is what unlocks simulation capability entirely.

This is a binary capability difference: without federation, the PolicySimulator returns empty trajectories for all shocks. With federation, it propagates shocks with 91% directional coherence.

6. C1 — Non-IID Verification

Method: Jensen-Shannon Divergence (JSD) between each country pair's empirical distribution per indicator. JSD ∈ [0, 0.5]; >0.3 = non-IID; <0.1 = near-IID.

Most heterogeneous indicators (JSD = 0.5, maximum possible):

Verdict (C1 confirmed): 49% of indicator pairs are maximally non-IID. This satisfies the FL prerequisite. Without this, federation could not be justified as solving a fundamentally harder problem than centralised learning.

7. Q2 — Online vs Batch (Characterisation, Not a Core Claim)

Online outperforms batch in 6/84 folds (7%). The justification for the online engine is not prediction performance — it operates in streaming mode without future look-ahead, and its hypothesis confidence evolves in real time. The 7% win rate is reported honestly.

8. S1 — Meta-Learning: Warm-Start Sensitivity

The non-monotonic curve is real: 5–10 cross-domain rows injected before local priors stabilise introduces noise that takes ~10 local steps to resolve. Benefit becomes persistent at 30 pioneers. This matches REPTILE/MAML behaviour: minimal but sufficient foreign-task initialisation outperforms no initialisation, but the warm-up window matters.

9. C2 — FL Justification: When Does Federation Help?

Cross-over point: 13 years of local data. Below this, federation adds noise faster than signal. The _not_ready() sentinel in the engine quantifies this empirically.

vs FedAvg: FedAvg's failure (MAE 0.687 vs Local 0.535) is structural, not tuning. Even at 100% data availability, parameter averaging creates models wrong for all countries. Scarcity's evidence-sharing avoids this: each node decides what to believe from peer data rather than having peer parameters imposed on it.

10. S2 — Ethiopia: Generalisation to Unseen Domain

The +29% warm-start advantage reflects structural patterns (inflation–interest linkages, debt–GDP relationships) that transfer across East African economies even when specific magnitudes differ. The GlobalMetaMemory provides portable initialisation that accelerates confidence accumulation in an unseen domain.

11. S3 — DRG: Compute Budget vs Discovery Quality

A node with 20× less memory achieves 94% of maximum confidence — graceful degradation. The trade-off is modest at this stream length and expected to be more pronounced at daily frequency.

12. C3 — Data Scarcity Curve

Confidence is positive at 8 years. The non-monotonic curve (trough at 20 years) reflects active exploration at 12–20 years, generating more hypotheses than can be confirmed. Recovery from 20–34 years is the confirmation phase.