This is the basic idea behind Goodhart’s Law:
“When a measure becomes a target,
it is no longer a good measure.”
it is no longer a good measure.”
Image Attribution: Jono Hey - sketchplanations.com (direct link)
The idea here is that once we make a measurable goal to
evaluate a more complicated process, people try to optimize for the specific
goal, which often undermines the actual larger objective. So schools end up “teaching to the test,”
scientists start “p-hacking,”[1]
academics form citation cartels to bolster their citation scores,[2]
and effective altruists end up choosing projects based not-so-much on
maximizing good, but maximizing measurable benefit.[3]
It should not surprise us that tying performance to
extrinsic motivators, attracts extrinsic motivation.
But much of the most-difficult value added to the world is intrinsically
motivated. And intrinsic motivation, is
cultivated in human-scale, character-forming communities.
I think a lot about Goodhart’s law.
I simultaneously try to utilize it and resist it. I try to produce units of value for people
who fund my work[4]
that are easily measured. But I also
actively resist the tyranny of measurement and try to carefully add unmeasurable value to my organizations and communities, because those intrinsic
investments keep institutions and communities healthy. And frankly, the former gives me capital for
the latter.
But I was recently on a podcast[5]
about the Christological themes of the Hebrew Scriptures, and suddenly found
myself citing Goodart’s Law in that conversation. I was probably as surprised as my
conversation partners. It was a new idea
for me and I was working it out in real time.[6] So I wanted to give it a little more careful
treatment here.
So here’s the question I was responding to:
Why do Christological prophecies sometimes seem a little
vague?
This is a common and legitimate knock against any sort of
prediction.[7]
If you keep it vague, multiple futures can fulfill it.
This was the objection I raised with my friends in Junior
High School when they told me that Nostradamus had predicted the immanent end
of the world and had a great track record of predicting historical events. And I join my 8th grade self in
acknowledging this liability of ancient and even Biblical prophecy.[8]
The specificity of the Messianic themes in the Hebrew
Scriptures are one of their defining qualities.
The Messiah will be a prophet[9]-king[10]-priest[11]
that comes from David’s family who will somehow suffer redemptively for God’s
people[12]
but also transcend that suffering as a cosmic figure who receives worship[13]
and everlasting dominion. This is a
paradoxical character that is both defeated and victorious, terrestrial and
transcendent, primate and deity.
The degree of difficulty here is outrageous. The messianic themes are so diverse and impose
so many, seemingly contradictory constraints on this future servant-sovereign,
that it would be very difficult to imagine one individual embodying them
all.
The degree of difficulty to qualify as messiah is really
high here…which is really important.
You see, these themes were so clear in the Hebrew tradition
that there were a bunch of dudes claiming to be the Messiah.[14] And this is where Goodhard’s Law comes
in.
Prophetic detail can be surprisingly unhelpful in validating
prophecy.[15]
Or to restate Goodhart’s Law for ancient prophecy:
“Attainable[16] detail in a prophecy
makes a poor metric for evaluating that prophecy’s fulfillment.”
If the ancient poems laid out a lot of situational details
identifying the Messiah, you can believe that aspiring Messiah’s would craft
their personal narratives and ret-con their biographies to check those boxes.
So the mundane technical detail that we think we want from
ancient prophecy to make it more evidential[17]
are not as valuable as they seem. What
makes ancient prophecy more convincing is a paradoxical degree of
difficulty.
A prophetic tradition with an outrageous degree of
difficulty, that places its validating criteria on the level of big paradoxical
claims, rather than a checklist of specific claims that become vulnerable to
Goodhart’s Law.
In light of the Goodhart-ian failure mode, the level of
specificity of the Christological prophetic tradition might be closer to optimal
than it might seem.
This post was written to the Run
River North channel on Pandora.
[3]
Which can leave more effective investments on the table if their results are
difficult to measure.
[4] My
#1 principle of professional science business model is “figure out what summary
statistics the people who give you money put on their year-end-review power
point slides…and then make a bunch of those things.” This means I spend a little more time that I
think is useful grinding through peer review (an opaque, erratic, and exploitative institution…that is also necessary in lieu of an alternative) than
its value justifies when a summary Youtube video would take 1/30th
of the time and be more broadly helpful.
[5]
Spotify: tinyurl.com/yourpodandyourstaff Episode 5 (I'm on several of the episodes, but 5 is the one on this topic).
[6]
This is what I like about podcasts. You
get unvarnished, novel ideas in conversation.
But it also means podcasts require more grace, because 90% of my
unvarnished, novel ideas are terrible.
[7]
Including scientific prediction. While we
can establish quantitative criteria to test and falsify scientific prediction,
those criteria are arbitrary. A p-value
of 0.05, is really just a statement of someone’s emotional tolerance for
uncertainty and false positives.
[8] It
is worth noting, that most biblical prophecy is not future-telling. And most of the future-telling biblical
prophecy is relatively-near future (about the immanent Assyrian/Babylonian
exile or the return from the latter). But there is a genre of Hebrew poem that
sees past the immanent catastrophe or deliverance to a future age and a future
figure, that is clear, compelling, and cohesive enough that contemporary
readers expected a Messianic figure who would embody these literary threads.
[9]
Deuteronomy 18:15-19
[10] 2
Samuel 7:16, Ezekiel 37:24-26
[11]
Zechariah 6:12-13
[12]
Isaiah 53
[13]
Matthew 26:62-65,
[14]
The Maccabean history is a pretty critical backdrop of the gospel narratives
(as I mention on the podcast). Israel
actually threw off their Hellenistic Hegemon for a brief period of independence
before Roman oppression, so the potential for this long awaited leader to free
them again was in the forefront of the political imagination.
[15]
One case I make is that using these prophecies to “prove” Jesus’ legitimacy or
the credibility of the Christian story diverges from their intent. Sure, they are faith building. But they were never meant to be faith
confirming. Trying to press them for
proof (or even evidence) is a post-enlightenment bias. The Christological themes are not primarily a
source of evidence…they are primarily a source of beauty.
[16]
When the Biblical record does provide biographical detail, it is the sort of
thing that would be difficult to stage: like ancestry or place of birth.
[17]
Which, again, isn’t the point. The point
is that we see that God was at the same game all along and that we are players
in a story who’s author knew the sublime climax and littered clues throughout
the text, starting on page 1.
