Wednesday, April 29, 2020

Goodhart’s Law and Optimal Prophetic Specificity


 Measuring results to motivate desired outcomes can generate unintended consequences and perverse incentives. 

This is the basic idea behind Goodhart’s Law:

“When a measure becomes a target, 
it is no longer a good measure.”

Image Attribution: Jono Hey - sketchplanations.com (direct link)

The idea here is that once we make a measurable goal to evaluate a more complicated process, people try to optimize for the specific goal, which often undermines the actual larger objective.  So schools end up “teaching to the test,” scientists start “p-hacking,”[1] academics form citation cartels to bolster their citation scores,[2] and effective altruists end up choosing projects based not-so-much on maximizing good, but maximizing measurable benefit.[3]

It should not surprise us that tying performance to extrinsic motivators, attracts extrinsic motivation.

But much of the most-difficult value added to the world is intrinsically motivated.  And intrinsic motivation, is cultivated in human-scale, character-forming communities.

I think a lot about Goodhart’s law. 

I simultaneously try to utilize it and resist it.  I try to produce units of value for people who fund my work[4] that are easily measured.  But I also actively resist the tyranny of measurement and try to carefully add unmeasurable value to my organizations and communities, because those intrinsic investments keep institutions and communities healthy.  And frankly, the former gives me capital for the latter.

But I was recently on a podcast[5] about the Christological themes of the Hebrew Scriptures, and suddenly found myself citing Goodart’s Law in that conversation.  I was probably as surprised as my conversation partners.  It was a new idea for me and I was working it out in real time.[6]  So I wanted to give it a little more careful treatment here.

So here’s the question I was responding to:

Why do Christological prophecies sometimes seem a little vague?

This is a common and legitimate knock against any sort of prediction.[7] 

If you keep it vague, multiple futures can fulfill it. 

This was the objection I raised with my friends in Junior High School when they told me that Nostradamus had predicted the immanent end of the world and had a great track record of predicting historical events.  And I join my 8th grade self in acknowledging this liability of ancient and even Biblical prophecy.[8]

The specificity of the Messianic themes in the Hebrew Scriptures are one of their defining qualities. 

The Messiah will be a prophet[9]-king[10]-priest[11] that comes from David’s family who will somehow suffer redemptively for God’s people[12] but also transcend that suffering as a cosmic figure who receives worship[13] and everlasting dominion.  This is a paradoxical character that is both defeated and victorious, terrestrial and transcendent, primate and deity.

The degree of difficulty here is outrageous.  The messianic themes are so diverse and impose so many, seemingly contradictory constraints on this future servant-sovereign, that it would be very difficult to imagine one individual embodying them all. 

The degree of difficulty to qualify as messiah is really high here…which is really important.
You see, these themes were so clear in the Hebrew tradition that there were a bunch of dudes claiming to be the Messiah.[14]  And this is where Goodhard’s Law comes in. 
Prophetic detail can be surprisingly unhelpful in validating prophecy.[15]
Or to restate Goodhart’s Law for ancient prophecy:

“Attainable[16] detail in a prophecy makes a poor metric for evaluating that prophecy’s fulfillment.”

If the ancient poems laid out a lot of situational details identifying the Messiah, you can believe that aspiring Messiah’s would craft their personal narratives and ret-con their biographies to check those boxes.

So the mundane technical detail that we think we want from ancient prophecy to make it more evidential[17] are not as valuable as they seem.  What makes ancient prophecy more convincing is a paradoxical degree of difficulty. 

A prophetic tradition with an outrageous degree of difficulty, that places its validating criteria on the level of big paradoxical claims, rather than a checklist of specific claims that become vulnerable to Goodhart’s Law.

In light of the Goodhart-ian failure mode, the level of specificity of the Christological prophetic tradition might be closer to optimal than it might seem.

This post was written to the Run River North channel on Pandora.




[3] Which can leave more effective investments on the table if their results are difficult to measure.
[4] My #1 principle of professional science business model is “figure out what summary statistics the people who give you money put on their year-end-review power point slides…and then make a bunch of those things.”   This means I spend a little more time that I think is useful grinding through peer review (an opaque, erratic, and exploitative institution…that is also necessary in lieu of an alternative) than its value justifies when a summary Youtube video would take 1/30th of the time and be more broadly helpful.
[5] Spotify: tinyurl.com/yourpodandyourstaff  Episode 5 (I'm on several of the episodes, but 5 is the one on this topic).
[6] This is what I like about podcasts.  You get unvarnished, novel ideas in conversation.  But it also means podcasts require more grace, because 90% of my unvarnished, novel ideas are terrible.
[7] Including scientific prediction.  While we can establish quantitative criteria to test and falsify scientific prediction, those criteria are arbitrary.  A p-value of 0.05, is really just a statement of someone’s emotional tolerance for uncertainty and false positives.
[8] It is worth noting, that most biblical prophecy is not future-telling.  And most of the future-telling biblical prophecy is relatively-near future (about the immanent Assyrian/Babylonian exile or the return from the latter). But there is a genre of Hebrew poem that sees past the immanent catastrophe or deliverance to a future age and a future figure, that is clear, compelling, and cohesive enough that contemporary readers expected a Messianic figure who would embody these literary threads.
[9] Deuteronomy 18:15-19
[10] 2 Samuel 7:16, Ezekiel 37:24-26
[11] Zechariah 6:12-13
[12] Isaiah 53
[13] Matthew 26:62-65,
[14] The Maccabean history is a pretty critical backdrop of the gospel narratives (as I mention on the podcast).  Israel actually threw off their Hellenistic Hegemon for a brief period of independence before Roman oppression, so the potential for this long awaited leader to free them again was in the forefront of the political imagination.
[15] One case I make is that using these prophecies to “prove” Jesus’ legitimacy or the credibility of the Christian story diverges from their intent.  Sure, they are faith building.  But they were never meant to be faith confirming.  Trying to press them for proof (or even evidence) is a post-enlightenment bias.  The Christological themes are not primarily a source of evidence…they are primarily a source of beauty. 
[16] When the Biblical record does provide biographical detail, it is the sort of thing that would be difficult to stage: like ancestry or place of birth. 
[17] Which, again, isn’t the point.  The point is that we see that God was at the same game all along and that we are players in a story who’s author knew the sublime climax and littered clues throughout the text, starting on page 1.

My Youtube Channel

I've been developing YouTube content lately.  It is an unsurprisingly eclectic station.   I'm working on some philosophical and theological content, but for now there are two main types of videos:

1. Soccer (Football) Tactics Videos

I realized a few years back that I go through multi-year obsession cycles.  At any given time, there is one skill that I invest huge hours in.  This decade it has been preaching, writing novels and short fiction, and for the last couple years...coaching.[1]

I send out a technique or tactics video every week on the training theme.  While there is a ton of great technique content[2]...tactical content has two main problems.

First: It is too advanced.  Most tactical videos were developed for sophisticated Premier League fans.  Not for 7-14 year olds who do not watch sports regularly.

Second: It always uses footage from the men's game.  But I mostly coach girls.  So I'd like to use content from the women's game.   So I've been developing some of these tactical videos with USWNT and NWSL footage:



2. Sediment Modeling Videos

I don't talk much about my vocation in this space, but I'm on a team that develops popular, hydraulic, modeling software.  Our user manual is excellent, and well over 1,000 pages.  The problem is, no one reads user manuals any more.  If I need help with a new program, I go to YouTube like a civilized person.  And sediment transport modeling - like many highly parameterized earth science simulations - is more of a wisdom-based craft than a hard skill.  So I'm trying to develop a series of videos that transmits the skill and wisdom components of the craft.


____________

[1] I may do a coaching manifesto at some point.  But the top tenant of a document like that would be the same as any of my other obsessions: "Embrace your incompetence."  Realizing that simply being a parent (and I think dad's are particularly prone to undue confidence here) who played the sport gets you about 20-30% to where I want to be.  Coaching is a skill, like other skills, that needs to be taught, learned, and practiced.  But it is the strange skill of transmitting skills...as well as character and tactics.  Of the three, the third is the easiest.

[2] I've developed a playlist of the videos I use most seasons: https://www.youtube.com/playlist?list=PL26VoPqZfrgBnXZP_4Z7erNrIxyZMX-kj

Thursday, May 10, 2018

ngram Shenanigans

Despite growing concerns about the attention economy[1], I love two Google innovations with untainted passion: Google maps and the ngram.[2]

You see, since Google has all the books - like, seriously, all of them - they are in a unique position for cultural data mining.  The beauty of the ngram is that it exposes the capability.  The ngram allows users to test historical hypotheses about cultural trends by computing time series of word occurrence.

In other words, you can plot the relative-frequency of terms or vocabulary through two centuries of texts and then make wild speculations[3] about the cultural trends behind those plots. 

For example, vampires have always been more popular than zombies:

There was a brief, single decade spike in vampire interest in the mid 19th century…and a recent phenomena that blew them up…though I can’t imagine what that would be.

Zombies on the other hand are a twentieth century monster, rising up in the wake of each world war.[4]  But the thing I like about the Zombie plot is that it is slowly and steadily creeping up…you know…like a zombie.

In a future post[5] I will use these to test some linguistic hypotheses that Robert Nisbet made in 1953.  But this is not that post…this post is the confluence of the many interests explored in this space, and this analytical tool.

So now that you know how these work, here are a few more.

__________________________

First, I often use this one when I explain my work at conferences or public meetings:


Dam building in Europe and North America peaked in the early 20-th century.[6]  But as they built at the best sites and our understanding of the ecological impacts grew, we stopped.  But dams also trap sediment, which fills the reservoirs.  So, about 80 years after the dam peak, interest in sediment follows a similar trend.

__________________________

I know that the name “Stanford” peaked in the 20’s and the railroad magnate is in play here and the time series is confounded by the university…but I can’t parse the strands of those influences here:

__________________________

I don’t feel like I need to add commentary here...

__________________

Also…without further comment.


…and rescaling contentment:

________________

The ngram also weighs in on the religious illiteracy of our information institutions.  They use words like “evangelical” and “fundamentalist” as if they were interchangeable[7] and recent, with self-evident referents.  I’d suggest that any pundit who can’t explain the time series in this plot should probably educate themselves a little before the start throwing around religious labels:

Though, I suppose it would be fair to counter that many American Christians don’t know our heritage enough to parse these threads either.

______________

All the major theological vocabulary are down. Church and worship follow the same trend on larger scales.
This should surprise no one, given secularizing trends and that devotional and theological texts composed a higher proportion of literature in the nineteenth century.  But a couple theological terms work against the trend, suggesting that certain emphases of twentieth century American Christianity are novel[8], and maybe not native to our belief system.



_______________

Speaking of innovations, I’d be lying if I said I expected this:

_______________________

But this was pretty predictable[9]



…as was this[10]


_______________________

… lightning round…



__________

And finally…


This post was “written” to the soundtrack of the Civil Wars Pandora Station



[1] I may do a post on this…and baseball…but for now:  I’ve been influenced a lot by Simone Weil who advocated for dull activities in our educational process for two reasons:  1) learning to sustain attention* is far more important than acquiring content for children and young adults.**  Everything good comes from the compounding interest on sustained attention.  But, more importantly, 2) attention is the raw material of love, which makes it the active ingredient in relationships and worship. 

So, an industry that competes for attention is a particularly pernicious “tragedy of the commons,” monetizing our most important resource.  And yes, I have switched my phone to black and white permanently.  Try it for one day and see if it doesn’t divest that thing of some of its illicit power.

But this isn’t my technology lament…this is a celebration of some of the beauty that can also come from that ferment.

*my favorite Weil quote goes something like “attention is the purest form of generosity”
**and not-so-young adults if our own educational experience did not help us train our attention

[2] Both of these are underrated.  I realize it is difficult to believe that google maps are underrated…but adding historical maps makes maps an unprecedented tool for fluvial geomorphologists and forensic contaminant engineers (fields I work or have worked in).  But I cannot believe I haven’t seen more ngrams in TED talks or tweets or other forms of multi-sensory speech.

[3] These trends are confounded by many data artifacts, mostly sample size non-stationarity.  Random preservation bias and directional preservation bias affect the early record disproportionately.  I’d expect both variability and uncertainty to be high earlier in the time series and potentially skewed by surviving texts…and also texts Googlers* deemed worthy of pursuing.

*I have so many thoughts about this word since I just finished Lazlo Bock’s book.  But I’ll leave them on the table.

[4] Born out of the ashes of the expectation of monotonic progress and the false hope of pax-tencologica.  Zombies are dystopic in a way vampires aren’t, and often have a culture-scale morality-tale built into their narratives, while vampires feature individual-scale morality-tales.

[5] Which I can only promise because it started as part of this one and is complete.

[6] For the one or two folks who are hydro-minded…the joke that always works here in water crowds is something like…”the dam time series is shaped like a hydrograph, with a steep rising limb and a gradual receding limb…and the sediment time series is the same hydrograph, with an 80 year translation time.”  I know…I’m hilarious.

[7] And even religion non-specific.  “Fundamentalism” is a specific movement of early 20-th century American Christianity, but the religious categories of our information economy are such blunt instruments that it gets foisted on Muslims in the Middle East and Iran and Hindus in India.  

[8] Like Jazz and baseball dispensationalism is an American innovation and export.

[9] Happiness follows the same trend as joy, just steeper…but I find joy to be a much more useful idea, and frankly, was surprised at the happiness trend.

[10] “Being unable to cure death, wretchedness and ignorance, men have decided, in order to be happy, not to think about such things.”  Pascal - Penses

Tuesday, October 17, 2017

My Top 10 Study Tips

This week I’m giving a talk on cultivating ‘tenacious curiosity,’ which makes a case from the book of Daniel that studying hard, even when the work is tedious or dull, can be part of Jesus following, God loving, faithfulness. 


It may or may not have included a clip from everyone's favorite song about Ashurbanipal.


The talk focused on the ‘why’, of taking studies seriously, but did not expand on the ‘how’.


I have no formal training on pedagogy.   But I have studied a lot, so I thought I’d pass along the results of my anecdotal experience.


Here are my thoughts on how to discipline curiosity and thrive academically in undergrad:


1.  Understand the Concept of Flow and the Value of Momentum.


Momentum is the currency of all creative work.  Whether you are writing a paper, or solving a problem set, or generating a lab report, or learning a language, 90, consecutive, undistracted minutes are much more valuable than 90 minutes interspersed with 30, or 60, or 90 minutes of ‘study breaks.’  


Creativity isn’t magical or mysterious.  Our brains are wonderful and even mystical organs, but they can be trained.  Creativity inhabits a mental space psychologists call ‘flow.’ The key to productivity in and after college is to learn how to create and sustain flow.


2. Create Sustained Undistracted Time Windows


The environment for flow is a substantial, but tractable, block of undistracted time.[1]  I think for most people it generally works best if it is in a regular, predictable time and space.   


Predictability and structure may seem like the opposite of the conditions for creativity, but creativity is a discipline not an emotion.  


If you learn to create a habitat for creativity, it will become a regular companion.  Buy a timer,[2] set it for 60 minutes at first, then 90, then 120 and, don’t go online or check e-mail.   See if you don’t learn to create and sustain momentum.


3. Track Your Time and Budget Your Hours


How many hours a week do you study?  If you don’t know the answer to that question ±3 hours you are probably working reactively rather than proactively.  Time is a commodity, and if you are going to spend it wisely, you need to budget it.  Here’s my recommendation:


i. Lay out a week in your favorite medium (graph paper, calendar, Excel, dragon skin and the blood of a newt,[3] medium doesn’t really matter).[4] 


ii. Subtract a healthy amount of sleep. If I was to make one change in the many regrettable study practices of my undergrad, I’d sleep more.[5] 


iii. Then block out the non-negotiable activities that are ‘above the line,’ activities that you prioritize before academics.  For me, in undergrad, it was church, participation in Intervarsity large and small groups, and, in my senior year, a Saturday date with my girlfriend.[6]  For some this will include exercise, close friendships, a sport or club.  With the rising costs of school, for more and more students, this includes non-trivial hours, working to pay the bills.  If video games or television are non-negotiable, budget a reasonable (and I would argue, a reasonably brief) time for them, and then stick to the budget.  Sabbath is a good principle here.  Make time for rest[7] and protect it. 


iv. Then find the blocks of time for work and study.  Mark some in stone as non-negotiable, the same time and place where you are not likely to be interrupted by friends or internet, and make others flexible in time and place, experimenting with what is most conducive to flow…but assess that critically and don’t self-deceive.[8]




v. Whatever you are doing, be fully present.[9]  Don’t watch Netflix during study time.  Don’t worry about the test while you are enjoying time with friends.  Just like with money, if you make a budget and stick to it, it not only keeps you from wasting the resource, it makes enjoying the part set aside for fun guilt free.



4. Find a Study Group for Problem Set Classes


This one is particular to STEM (Science Technology Engineering and Math).  A good study group can be hit or miss in the humanities, social sciences and even the descriptive sciences.  They can sometimes waste more time than they save for reading, essay exam, discussion, and term paper based classes.  But for classes based on problem sets (Math, Physics, Chem…) find a study group, or make one.  The best study groups meet at the same time each week to tackle a regular problem set, early enough before it is due so that you can each visit office hours to get help on problems that you get stuck on.


It is ideal if there is someone else in the group as good or better than you at problem solving.  Often if there is at least someone of your caliber in the group, you will figure out different problems or different parts of the same problem, saving time.  But a study group is valuable in STEM even if you are the strongest performer in the group.  As long as the others don’t simply copy your work, if they solve the same problem even with your approach, they can verify your solution independently (or catch mistakes).


5. Go to Class…Always…and Early


Class is the best use of your study time.  There is no better return-on-investment for your time than going to lecture.  The time cost of reconstructing the material from a class is almost always much too high to make missing  class worth it, even if you have a test in your next class.
Also, if you get to class 5-10 minutes early and review the previous class’ notes, you will get more than 5-10 minutes of value from that time, because it is close enough to the first introduction to plant it more firmly in your memory, and because it will help you understand the new material (that will, presumably, build on it).


6. Notes > Text


This might not be true for English, but in all the STEM classes I took, and most of the liberal arts classes, the professor was far more interested in what he or she told me than what the book had to say.  In many cases the professor has forgotten what was in the book.  I have a very close friend whose GPA jumped a full point when she stopped spending all her time reading the text and really metabolized the notes. 

Granted, this is a tip for grad chasing rather than learning, but it is best if your learning manifests as grades, and your time is limited.


And while we are talking about grade optimization rather than learning, the best thing you can do for your grades is to read the syllabus, carefully, paying attention to the percentages, and apportioning your effort accordingly.


7. Stop Telling Yourself You Work Best Under Pressure…Seriously, Just Stop


“I work best under pressure” is self deception.  Work under a deadline is rushed and unreflective.  What you are experiencing ‘counterfeit flow’ is a truncated version creative productivity that comes from finally giving yourself a small, sad, exhausted block of undistracted time.  But you would have had the same experience, only sharper and more productive and more creative, if you had given yourself the same block of time a week earlier, in the morning or mid-day.  


Necessity is the friend of productivity but the enemy of creativity.


You might work fastest under pressure, but real learning requires time to reflect, and creativity is a middle-distance race not a sprint.[10]  You are most productive when you capture a state of flow (see #1) which usually requires regular, predictable setting of undistracted time.


8. Work While You Aren’t Working…By Working Ahead


Most creative work happens when you aren’t actively working.  While you are eating or exercising or at a job or talking with friends…shoot, I once dreamed the solution to a physics problem.  But you can only take advantage of the ‘work when you aren’t working’ phenomenon if you start early enough to have the basic ideas you are working on in the cognitive hopper for a while before the deadline.[11]


If you have a paper outline in your head, you will fill it out while you walk to class, then when you sit down to work, momentum is close at hand.


9. Ask Yourself “Why Did my Professor Find this Subject Interesting Enough to Dedicate her Life To It?” and Then Try to Contract the Virus.


Your professor dedicated her life to this topic.  At one time or another she found it that inspiring.  Surely there is enough beauty and value in there for you to squeeze 4 quarter credits of attention out of it.  And if your professor is seriously done, like out to tenure pasture done, long past inspiring anyone, then try to figure out what is so enchanting about the topic that your TA is committing her future to it.


You don’t have to make the topic your vocation, but you’ll have a lot more energy for the quarter if you find the secret to their passion.


10. Experiment with Academic Generosity.


I’m convinced that generosity is the most powerful prescription for worry and discontent.[12]  The problem is that we generally think of generosity in terms of money, and undergrads have none of that…in fact, given the loan situation, most undergrads actually have negative money.


But the most precious resources in undergrad are time and GPA. 


Experimenting with academic generosity, giving time and donating insight when your time and insight can make the difference for someone else.  This will deconstruct the competitive power of the academic life.


So that is my eccentric, anecdotal top 10.   


I also reached out to some of my professor friends to ask them.  I posted my friend Prof. Rich's, Prof. Teresa's, and Prof. George's thoughts in the comments and rolled Prof. Tiffany's thoughts into a second post here.




This post was written while listening to In the Aeroplane Over The Sea[14] by Neutral Milk Hotel


[1] I have a friend who used to work in increments of ‘one laptop battery’ (which used to be just over two hours).  For him, that enough time to achieve flow, but stopped  for a break before the point of diminishing returns.
[2] Like a real, life, physical timer.  Don’t rely on an internet timer, because you want to be going to the internet as little as possible.
[3] Actually, dragons are apex predators, so even in an imaginary world, commoditizing their pelt is probably morally dubious, and if you are handling dead newts, be aware that the contain one of the most toxic neruo toxins in the animal kingdom.  Maybe better to go with a moleskin and a pen…or at least sheep hide and frog blood.
[4] I use software for this, but I’ve had friends tell me that is weird
[5] The sleep deficit I ran made me exhausted in class, so even though I was there, I had to re-learn the material and had horrible notes, and when I tried to work, I was too tired to be creative.  It is a psychological reality.  You are more creative when you are well rested.  Red bull does not provide the same brain chemicals as sleep…espresso does not consolidate and reinforce neural pathways.  Sleep is part of learning.
[6] My now, wife.  This was the best investment I made in undergrad.
[7] Rest gets confused with as entertainment.  In your very limited non-work time, make sure to actively choose activities that are life giving.
[8] If you don’t have enough time in iv, revisit iii and your definition of ‘non-negotiable,’ or lower your academic expectations and be cognoscente of the future costs of that decision.
[9] This was my friend Tiffany’s approach to her first year of Grad School at a top 5 English program (where she measured the reading she had to do that year in meters).  I stole it, and it has served me well.
[10] I think the ‘marathon-not-a-sprint’ metaphor is overused and mostly inaccurate.  If an all nighter is a sprint, by simple temporal upscaling, writing a paper carefully and reflectively over the course of 2 weeks is a middle distance race, something like an 800m or a 1600m.  Nothing in the quarter system is a marathon, but it’s also better if it’s not a series of sprints either.
[11] This may seem to contradict my advice to ‘be fully wherever you are’ but it doesn’t really.  The ‘be fully present’ principle is about worry and distraction.  If you have learned why your topic is fundamentally interesting or even beautiful, thinking about it recreationally does not violate the boundaries of Sabbath.
[12] I actually did a talk about this.
[13] I have reached out to some of my professor friends to provide a few thoughts of their own.
[14] It was also written IN an aroplane, somewhere over Nevada.