Coaching Letter #242

Hi there, I hope you are doing well, and thank you for subscribing to The Coaching Letter. You are amazing. This Coaching Letter concerns the challenge of trying to discern and preserve the complexities that comprise the science of learning (or anything else, actually) as we currently understand it. This topic was prompted by my encountering several pieces lately that flatten many of these complexities in such a way that the science seems simpler and more clear-cut than is actually the case—which, for the new subscribers and those too busy to read it—was the topic of the last Coaching Letter. I wasn’t really terribly happy with that Coaching Letter, so this is an attempt to put some framing around it that summarizes what I was trying to say.

TL;DR: Hedgehogs organize the world around one big idea; foxes draw on many ideas, tolerate ambiguity, and revise their thinking as circumstances change. The distinction provides a useful frame for understanding how people interpret and use research: hedgehogs turn findings into universal prescriptions, while foxes preserve the complexity needed to apply them wisely. Educational research rarely tells us simply “what works.” Good decisions require asking what works, for whom, under what conditions, and at what cost—which means attending to boundary conditions, confounding variables, and opportunity costs; testing our causal assumptions; and updating our beliefs as evidence accumulates.

I think there is a helpful overarching concept, represented by the distinction between foxes and hedgehogs. I learned this from Philip Tetlock’s book, Superforecasters (2015), but the idea is much older (from Archilocus, a 7th century BCE Greek poet). The idea is that the hedgehog knows one big thing, but the fox knows many things. According to Isaiah Berlin (1935) the hedgehog relates everything to one central view of the world, but the fox draws on many ideas that may not fit neatly together. Berlin preferred hedgehogs. But I like to think of the fox as wily, cunning, and dealing in subtlety and complication. In Nate Silver’s book, The Signal and the Noise, Silver uses the metaphor to distinguish between hedgehogs who only have one big idea, and so that’s what they talk about all the time—but because they keep saying the same thing over and over, they are better at grabbing attention. (think of TV pundits)—and foxes, on the other hand, who understand boundary conditions, confounding variables, and opportunity costs: not everything works in all situations, and understanding the contingencies that surround a finding is at least as important as knowing the finding itself.

I work at being a fox, but I’m not always good at it. I am prone to making grandiose claims when I get carried away by the particular idea I’m currently thinking about, and I am always grateful when I get called on it. (My colleagues are useful sources of corrective feedback, and I love them.) But at least I am working on being a fox. Lots of people lean into being hedgehogs. I presume that this is because they get more attention that way, so it’s reinforcing; and I can give you lots of examples of folks whose writing on platforms like this one includes claims that academic research doesn’t fully support—I fear my frustration and disparagement came across too strongly in CL #241—or was at least impolitic.

Anyway, the evidence suggests that foxes are better decision makers (Tetlock, 2005). Here are two sources on the distinction between hedgehogs and foxes, one from an author who uses that terminology and one who doesn’t, but whose examples are super useful…

Tetlock’s earlier research showed that pundits are no better than a coin toss when it comes to being right about what actually comes to pass, as compared with their predictions. Super-forecasters, on the other hand, are frequently right—that’s why he calls them super-forecasters. What Tetlock’s super-forecasters do:

• Break large, vague questions into smaller questions that can actually be investigated.
• Begin with base rates and the outside view, rather than treating every situation as unprecedented.
• Draw on several ideas and perspectives instead of forcing everything through one grand theory.
• Think in probabilities rather than certainties, and distinguish what is likely from what is merely possible.
• Update their beliefs frequently when new evidence arrives, even when the evidence points away from a favored conclusion.
• Actively look for information that could prove them wrong.
• Keep track of their predictions, learn from their errors, and work to bring their confidence into line with their accuracy.

Dörner does not use the fox-and-hedgehog metaphor, but the people who cope well with the complex systems simulated in The Logic of Failure (probably my favorite book of the summer; highly recommend) look distinctly fox-like. What Dörner’s successful problem-solvers do:

• Try to build a picture of the whole system (even if imperfect) rather than attending only to the most visible problem.
• Look for relationships among variables, including feedback loops, competing goals, and constraints.
• Pay attention to time delays and side effects, recognizing that an intervention can appear to work before its costs become visible.
• Make their assumptions explicit and test them against what actually happens.
• Use small, reversible interventions when the consequences of a large move are hard to predict.
• Monitor the effects of their actions and keep revising both their plans and their understanding of the system.
• Preserve room to maneuver instead of exhausting their resources in pursuit of one objective.

And when it comes to using research, the hedgehog v. fox distinction is between those who are interested in promulgating their one big idea (hedgehogs) and those who are interested in understanding the complications inherent in any research finding or educational setting (foxes). Here are other ways of thinking about those distinctions. In several sources, Dylan Wiliam (fox) makes the point that “Everything works somewhere, and nothing works everywhere. The right question is: ‘Under what conditions does this work?’ ” (See for example, this 2006 article on formative assessment,which also includes the idea of the “lethal mutation” that we have found super useful). And in Learning to Improve, the authors (foxes) state: “The critical issue is not what works, but rather what works, for whom and under what set of conditions.” In fact, that is what they label the second principle of improvement science: Variation in performance is the core problem to address. They are pointing out that asking “What works?” or “What is best?” are the wrong questions, because the answers may vary depending on any number of factors, including but limited to the skill and/or experience of the teacher, the background knowledge of the kid, psychological safety, the capacity of the district to support an initiative, the leaders’ stance towards improvement, the presence or absence of other initiatives competing for time and attention, the quality of the instructional materials, the make-up of the board, the success of a pilot, resources for implementation, how confident you are of success, and so on.

So here’s a short but powerful list of things that I don’t think hedgehogs understand (or choose to ignore, depending on how cynical you are):

Boundary conditions. I think I’ve written about boundary conditions before, but I’m not sure where… Maybe it’s just that I know I’ve written the following sentence, or one very like it, many times: The concept of boundary conditions has a mathematical origin that I don’t claim to understand¹. But I do know that by the late 1800s and early 1900s, philosophers of science (a catch-all term that predates the establishment of separate disciplines such as economics and anthropology) were aware that many “laws” or scientific principles cannot be separated from assumptions, context, and the vagaries of measurement. And the concept was formalized in an article by David Whetten in 1989, who argued that a theory needs to specify:

1. What factors are involved?
2. How are they related?
3. Why should those relationships hold?
4. Who, where, and when describe the outer limits of the theory’s range?

This is another way of stating the points made above: it’s not enough for a researcher to say that something works. They have to specify what claim they’re making (this causes that), under what conditions, for how long—and when does the principle/rule/theory no longer apply? For example, some bloggers claiming to espouse the “science of learning” (whatever that means) would have you believe that employing worked examples is a necessary feature of high quality math teaching. But a more accurate claim is (and I had ChatGPT help me with this so I wouldn’t get it wrong): Worked examples tend to benefit learners when the solution procedure is unfamiliar and imposes substantial demands on working memory; as learners acquire relevant schemas, the benefit diminishes and can reverse. That’s giving you the claim (worked examples are useful) but also describing the boundary conditions (when the learner is a novice with respect to the salient conceptual and procedural knowledge and NOT when the learner has sufficient knowledge to work the problem on their own). Foxes LOVE boundary conditions; hedgehogs not so much.

Confounding variables. These are variables which represent what’s really going on in a relationship, when it might be easy to be misled by a surface relationship. More specifically: “ A confounding variable is a third variable that is associated with both variables of interest and therefore distorts the interpretation of the relationship between the first two variables.” There are lots of examples of this which I’m sure you’ve heard already: shark attacks and ice-cream consumption are correlated, a relationship that can be explained by the fact that they both increase in the summer, not that ice cream consumption causes shark attacks; cities with more churches have more crime, not because religion causes crime but because larger cities have more of both. These seem funny and obvious, but in one of my previous lives I was a chief academic officer who had to show up to board meetings and sometimes give reports, and some of the questions I got were clearly about confounding variables. Here is my favorite example. We had a large number of charter schools in our district, as it was a charter-friendly, open-enrollment state. The student achievement was, on average, higher in the charter schools (but not the longitudinal growth, I hasten to add). The kids in the charter schools wore uniforms, and one of the board members grilled me on why we didn’t make uniforms compulsory for all students in the district, since kids in the charters wore uniforms and their scores were higher. And just in case it needs to be said, there were all kinds of ways in which the kids in charter schools were different from the non-charter population, including a much smaller percentage of students qualifying for special education services, and far fewer multi-lingual learners, and the charters didn’t provide transportation so that was another selection mechanism—kids in the charters had a parent able to drop them off and pick them up, which was likely related to other advantages they had over some other kids. But that board member had made their mind up that uniforms were the way to go… That board member could not, apparently, tell the difference between correlation and causation, and didn’t want to consider that there could be confounding variables that were more causal.

So when we look at a relationship we have to be willing to consider what else is going on. Just recently there was an article in Education Week about the “science of math” (whatever that is) that names that some teachers believe their students need more explicit instruction of the problem-first approach in the district-adopted math programs. (I’m accepting her characterization of these programs.) She appears to accept their diagnosis that sequence is the problem. But other information in the article also allows me to infer that neither she nor the teachers really understand the purpose or the practice of “problem-first” approaches, which means that it’s also possible that teachers have not had enough training or support to teach the program adequately, they may not have enough MKT (there’s a whole literature on Mathematical Knowledge for Teaching or MKT²), they may have assumptions about what their kids are capable of, they may conflate productive struggle with frustration… There may be all kinds of other things going on. The big problem is that, if there are confounding variables that are actually at the source of students’ lack of success, then switching from one math program to another, or teaching a program differently, is unlikely to bring about a significant change in student achievement.

In other words, if you don’t understand the causal relationships—which you won’t until you create some hypotheses and test them, and even then your theory is still definitely incomplete and possibly wrong (Bryk et al, 2015)—then you might very easily end up conditioning on the wrong variable, which I wrote about in CL #212. Also, CL #232 is about misattribution in scaling, which frequently entails confounding variables.

Opportunity costs. I also learned about opportunity costs from Dylan Wiliam; in pointing out that everything works under some condition or other, we also have to be cognizant that resources are limited, and choosing one course of action almost always closes the door on another. When we make decisions to do things like add co-teaching to the schedule, reduce class size, install SmartBoards, provide classroom aides, purchase comprehensive instructional materials (textbooks + teachers’ guides + student workbooks + teacher training), we have committed what often amounts to millions of dollars that are now unavailable for anything else. I think about this often when I see districts cycle through several instructional programs in a short period of time. Perhaps one is measurably better than the other. But is it so much better that it’ll be worth it in opportunity costs? Because those costs are not just in money. A new program also demands time and attention in terms of professional learning time, collaborative time, teacher capacity to learn new practices, and political and social capital. There are more examples related to this idea in CL #232, part of the series on Scaling. To bring it back to the math example, before reorganizing the math program, the fox would ask: Would the time and capacity required to change the instructional sequence produce more benefit than using those same resources to strengthen teachers’ MKT, task launches, monitoring, or consolidation?³

Please don’t take away from this that everything is so context-dependent that we can never make useful generalizations. “It depends” does not mean that research has nothing to contribute—far from it. Research can help answer questions that it might take an individual teacher an entire career to figure out alone. But we still do a poor job of teasing apart what works, for whom, and under what circumstances. I expect we will see a whole new array of hedgehogs wielding “the science of” this and “the science of” that coming towards us, and I want us to be well equipped, as foxes, to stand up to them.

If you have thoughts, I would love to hear them. Thanks for reading. Let me know if you need anything. Best, Isobel