LESSON 5: TESTING THE MODEL — HOW TO KNOW IF WE’RE WRONG (AND WHY THAT’S GOOD)

So far, we’ve built a map of society, made some rules about how it changes, and learned to speak in probabilities.

Now comes the most important — and humbling — part:
Testing our model to see if it actually works.

In science, a model isn’t true because it’s logical or beautiful.
It’s useful only if it can predict reality better than guessing.

1. THE “BACKTEST”: TRAVELING TO THE PAST

We can’t wait 10 years to see if our predictions come true.
Instead, we pretend we’re in the past and see if our model would have worked then.

The Backtest Recipe:

  1. Pick a past period: e.g., 1990–2010.
  2. Hide the future: Only use data up to, say, the year 2000.
  3. Run the model: Make a “prediction” for 2001–2010.
  4. Compare to reality: Did the predicted “probability clouds” match what actually happened?

Example:
If our model said in 1999:
“Country A has a 70% chance of civil war before 2010”
…and a civil war actually started in 2003…
…that’s a strong signal our model might be useful.

2. MEASURING SUCCESS: THE “FORECAST SCORECARD”

We don’t just say “it worked” or “it didn’t.”
We measure exactly how good our predictions were.

Tool 1: The Brier Score — Measuring Probability Accuracy

Imagine we forecast:

  • 80% chance of protests in Egypt in 2011
  • 20% chance of protests in Canada in 2011

What happened?

  • ✅ Egypt had protests.
  • ✅ Canada did not.

The Brier Score calculates how “calibrated” our probabilities were:

Brier Score = (forecast - outcome)²

For Egypt: (0.8 - 1)² = 0.04
For Canada: (0.2 - 0)² = 0.04
Average Brier Score = 0.04 (Perfect = 0, Worst = 1)

Lower score = better forecasts.

Tool 2: The Confusion Matrix — Counting Right & Wrong

We sort predictions into four boxes:

                  ACTUAL
               Crisis   No Crisis
PREDICTED
  Crisis       [HIT]   [FALSE ALARM]
No Crisis   [MISS]     [CORRECT QUIET]

A good model has:

  • Many HITS (predicted crisis → crisis happened)
  • Few MISSES (missed a real crisis)
  • Few FALSE ALARMS (cried wolf)

Tool 3: The AUC-ROC — Seeing the Trade-off

There’s always a trade-off:
If we predict “crisis” too often, we get more HITS but also more FALSE ALARMS.
If we rarely predict crisis, we get fewer FALSE ALARMS but more MISSES.

The AUC-ROC curve helps us find the sweet spot:

True Positive Rate (Hits)
    ↑
1.0 |     • Perfect Model
    |    /
    |   /
    |  /
    | • Random Guessing
0.0 └─────────────→ False Positive Rate (False Alarms)
    0.0            1.0
  • AUC = 1.0: Perfect model (never wrong)
  • AUC = 0.5: Useless (same as guessing)
  • AUC > 0.7: Decent
  • AUC > 0.8: Good
  • AUC > 0.9: Excellent

3. LEARNING FROM FAILURE: THE “POST-MORTEM”

When our model is wrong — and it will be — we don’t hide it.
We do a Post-Mortem to understand why.

Example: Our model failed to predict the Arab Spring in some countries.

Post-Mortem Questions:

  1. Missing Variable? Did we ignore social media’s viral spread?
  2. Wrong Threshold? Was our “unrest threshold” too high?
  3. Random Shock? Was it a true “black swan” no model could catch?
  4. Feedback Loop? Did leaders’ reactions change the outcome?

Then we update the model:

  • Add a new variable: “Social media penetration”
  • Adjust the threshold
  • Add a “leader response” sub-model

This is the scientific cycle:
Predict → Test → Learn → Improve.

4. THE DANGER OF “OVERFITTING”

There’s a trap: we could make a model that’s perfect for the past but useless for the future.

How? By adding too many rules tuned to past quirks.

Example:
If we add a rule: “Crisis always happens when the World Cup is in South America”
…it might fit past data…
…but it’s nonsense for predicting the future.

The Guardrail:
We test on completely unseen data — a period we never used to build the model.
If it works there too, it’s robust.

5. THE CMH MODEL PASSPORT

Every CMH model should come with a “passport” stating:

  1. Trained on: 1990–2010 data
  2. Tested on: 2011–2020 data (unseen!)
  3. AUC-ROC: 0.78
  4. Brier Score: 0.21
  5. Best at predicting: Regime transitions
  6. Worst at predicting: Sudden economic collapses
  7. Known blind spots: Small island nations, post-war states

This is transparency. It tells users exactly what they’re getting — and what they’re not.

YOUR CMH HOMEWORK:

  1. Backtest in Your Head: Think of a big event in your country’s last 20 years (election upset, protest, crisis). What three variables from Lesson 2 would have best predicted it?
  2. Score a Prediction: Last week’s weather forecast said “60% chance of rain.” Did it rain? Calculate its Brier Score:
  • Rain happened → Outcome = 1
  • No rain → Outcome = 0
    Formula: (0.6 - outcome)²
  1. Spot Overfitting: Look at any “pattern” in your daily life (e.g., “I always get a green light at Maple Street”). Is it a real rule or just a coincidence? How could you test it?

When you’re ready, we’ll move to Lesson 6: From Prediction to Action — The Ethics of Intervening in History.

This is where CMH stops being a forecasting game and becomes a tool for reducing human suffering — if used wisely.

Similar Posts