Goodhart’s Law and Optimal Prophetic Specificity

 Measuring results to motivate desired outcomes can generate unintended consequences and perverse incentives. 

This is the basic idea behind Goodhart’s Law:

“When a measure becomes a target, 
it is no longer a good measure.”

Image Attribution: Jono Hey - sketchplanations.com (direct link)

The idea here is that once we make a measurable goal to evaluate a more complicated process, people try to optimize for the specific goal, which often undermines the actual larger objective.  So schools end up “teaching to the test,” scientists start “p-hacking,”[1] academics form citation cartels to bolster their citation scores,[2] and effective altruists end up choosing projects based not-so-much on maximizing good, but maximizing measurable benefit.[3]

It should not surprise us that tying performance to extrinsic motivators, attracts extrinsic motivation.

But much of the most-difficult value added to the world is intrinsically motivated.  And intrinsic motivation, is cultivated in human-scale, character-forming communities.

I think a lot about Goodhart’s law. 

I simultaneously try to utilize it and resist it.  I try to produce units of value for people who fund my work[4] that are easily measured.  But I also actively resist the tyranny of measurement and try to carefully add unmeasurable value to my organizations and communities, because those intrinsic investments keep institutions and communities healthy.  And frankly, the former gives me capital for the latter.

But I was recently on a podcast[5] about the Christological themes of the Hebrew Scriptures, and suddenly found myself citing Goodart’s Law in that conversation.  I was probably as surprised as my conversation partners.  It was a new idea for me and I was working it out in real time.[6]  So I wanted to give it a little more careful treatment here.

So here’s the question I was responding to:

Why do Christological prophecies sometimes seem a little vague?

This is a common and legitimate knock against any sort of prediction.[7] 

If you keep it vague, multiple futures can fulfill it. 

This was the objection I raised with my friends in Junior High School when they told me that Nostradamus had predicted the immanent end of the world and had a great track record of predicting historical events.  And I join my 8^th grade self in acknowledging this liability of ancient and even Biblical prophecy.[8]

The specificity of the Messianic themes in the Hebrew Scriptures are one of their defining qualities. 

The Messiah will be a prophet[9]-king[10]-priest[11] that comes from David’s family who will somehow suffer redemptively for God’s people[12] but also transcend that suffering as a cosmic figure who receives worship[13] and everlasting dominion.  This is a paradoxical character that is both defeated and victorious, terrestrial and transcendent, primate and deity.

The degree of difficulty here is outrageous.  The messianic themes are so diverse and impose so many, seemingly contradictory constraints on this future servant-sovereign, that it would be very difficult to imagine one individual embodying them all. 

The degree of difficulty to qualify as messiah is really high here…which is really important.

You see, these themes were so clear in the Hebrew tradition that there were a bunch of dudes claiming to be the Messiah.[14]  And this is where Goodhard’s Law comes in. 

Prophetic detail can be surprisingly unhelpful in validating prophecy.[15]

Or to restate Goodhart’s Law for ancient prophecy:

“Attainable[16] detail in a prophecy makes a poor metric for evaluating that prophecy’s fulfillment.”

If the ancient poems laid out a lot of situational details identifying the Messiah, you can believe that aspiring Messiah’s would craft their personal narratives and ret-con their biographies to check those boxes.

So the mundane technical detail that we think we want from ancient prophecy to make it more evidential[17] are not as valuable as they seem.  What makes ancient prophecy more convincing is a paradoxical degree of difficulty. 

A prophetic tradition with an outrageous degree of difficulty, that places its validating criteria on the level of big paradoxical claims, rather than a checklist of specific claims that become vulnerable to Goodhart’s Law.

In light of the Goodhart-ian failure mode, the level of specificity of the Christological prophetic tradition might be closer to optimal than it might seem.

This post was written to the Run River North channel on Pandora.

---

[1] https://www.wired.com/story/were-all-p-hacking-now/

[2] https://www.nature.com/news/brazilian-citation-scheme-outed-1.13604

[3] Which can leave more effective investments on the table if their results are difficult to measure.

[4] My #1 principle of professional science business model is “figure out what summary statistics the people who give you money put on their year-end-review power point slides…and then make a bunch of those things.”   This means I spend a little more time that I think is useful grinding through peer review (an opaque, erratic, and exploitative institution…that is also necessary in lieu of an alternative) than its value justifies when a summary Youtube video would take 1/30^th of the time and be more broadly helpful.

[5] Spotify: tinyurl.com/yourpodandyourstaff  Episode 5 (I'm on several of the episodes, but 5 is the one on this topic).

Anchor: https://anchor.fm/s/1889f860/podcast/rss

[6] This is what I like about podcasts.  You get unvarnished, novel ideas in conversation.  But it also means podcasts require more grace, because 90% of my unvarnished, novel ideas are terrible.

[7] Including scientific prediction.  While we can establish quantitative criteria to test and falsify scientific prediction, those criteria are arbitrary.  A p-value of 0.05, is really just a statement of someone’s emotional tolerance for uncertainty and false positives.

[8] It is worth noting, that most biblical prophecy is not future-telling.  And most of the future-telling biblical prophecy is relatively-near future (about the immanent Assyrian/Babylonian exile or the return from the latter). But there is a genre of Hebrew poem that sees past the immanent catastrophe or deliverance to a future age and a future figure, that is clear, compelling, and cohesive enough that contemporary readers expected a Messianic figure who would embody these literary threads.

[9] Deuteronomy 18:15-19

[10] 2 Samuel 7:16, Ezekiel 37:24-26

[11] Zechariah 6:12-13

[12] Isaiah 53

[13] Matthew 26:62-65,

[14] The Maccabean history is a pretty critical backdrop of the gospel narratives (as I mention on the podcast).  Israel actually threw off their Hellenistic Hegemon for a brief period of independence before Roman oppression, so the potential for this long awaited leader to free them again was in the forefront of the political imagination.

[15] One case I make is that using these prophecies to “prove” Jesus’ legitimacy or the credibility of the Christian story diverges from their intent.  Sure, they are faith building.  But they were never meant to be faith confirming.  Trying to press them for proof (or even evidence) is a post-enlightenment bias.  The Christological themes are not primarily a source of evidence…they are primarily a source of beauty. 

[16] When the Biblical record does provide biographical detail, it is the sort of thing that would be difficult to stage: like ancestry or place of birth. 

[17] Which, again, isn’t the point.  The point is that we see that God was at the same game all along and that we are players in a story who’s author knew the sublime climax and littered clues throughout the text, starting on page 1.