June 11, 2024
My BFF Ann O’Doherty reading Chapter 7 of Equitable School Improvement
What Equitable School Improvement says about data…
Hello, how are you, thank you for subscribing to the Coaching Letter. You rock. If you are in the United States, you are just finishing up the school year, which I know is an incredibly hectic and tiring time. Good luck! And quick bit of advertising: if you are in Connecticut, or close by, we hope you will consider joining one of our communities of practice for next year: Equity, Diversity, Inclusion Network, Instructional Leadership Collective, and the High School Learning Collective. The first two are for individual leaders, and the third is for teams interested in focusing on improving high school instruction. And we have two online, free book study series available to you wherever you are: Systems for Instructional Improvement and Mathematics Tasks for the Thinking Classroom, Grades K-5. We hold these over Zoom in the afternoon Eastern time. We’re also hosting a book study for Peter Liljedahl’s new book, and Professor Liljedahl himself will speak at the first meeting on July 11, 4pm Eastern. Sign up here. Free and open to all.
I’ve been thinking about data a lot recently, for a variety of reasons, and then Tom Geraghty’s latest Psychological Safety newsletter is all about data, so I took that as a sign. Not incidentally, Tom’s newsletter is worth signing up for—you can do that from the link above. I’ve borrowed extensively from Tom—about psychological safety, about newsletter writing, and even the stickers I use draw heavily on his for inspiration. In education, we tend to pay the most attention to the metrics that everyone thinks matter the most, which are the standardized test scores. Rydell and I wrote a book that came out this year called Equitable School Improvement: The Critical Need for the Human Side of Change, which I hope you’ll read! Actually, I hope you’ll buy it for everyone on your team, all your friends, and your Great Aunt Matilda. There’s a chapter on data that takes issue with the way we think about and use data in K-12, so this CL is about the main points.
What the chapter is trying to do is reconcile the distrust of data among educators, many of whom have seen test scores be used to justify a bending of the curriculum towards tested subjects, leading to a thinner instructional experience for students with previous low test scores, who are disproportionately students of color; or they themselves have felt beaten up by data, which feels crummy when those outcomes are not totally under their control; or there is a culture of accountability that weakens bonds of trust precisely where they need to be strongest.
Can I just say that as I’m working on this, it’s really hard not just to copy big chunks of what we already wrote?
- Data are not equity neutral. There is a very long history of data being employed to justify racism and racist laws and policies. Data will be interpreted differently based on the mental models of those doing the interpretation, which is why more women are hired to play in orchestras when the auditions are blind. And even “objective” measures of data such as achievement tests are corrupted by social influences such as stereotype threat.
- Equity is not just about test scores. Our definition of equity includes access, opportunity, and experience as well as outcomes. We don’t want students constrained to feeble instruction commensurate with narrow curricular exposure in service of doing well on a test. That is, as my mother would say, cutting off your nose to spite your face. All students deserve a rich, world-building experience of school, and nothing less.
- Data in schools often seems to reinforce a deficit mindset. We hear comments about students not being “ready” for challenging math problems, for example, because they scored “at a fourth grade level” which educators seem to think means that they can do only 4th grade work. This is a massive misunderstanding. Where we have seen this show up lately—and this is not in the book—is in math, where teachers assume that any given problem requires prerequisite knowledge that may seem necessary from the perspective of a mathematics learning progression as reflected in the curriculum, but the problem can be approached in other ways that don’t require those prerequisites. A 6th grader scoring at a 4th grade level on a given test can still think like a 6th grader.
- Instead of thinking of data in terms of student and teacher performance, think in terms of the system that’s creating those outcomes. Everyone seems to have heard the term “Every system is perfectly designed to produce the results it gets”, but we are very bad at following through on the implications of that maxim. In almost all cases, people are doing the best they know how to do, and what they think you want them to do, so if that’s not creating the results we want, that’s a systems/strategy/process problem not a people problem.
- Wanting to hold people accountable for outcomes without caring about process is stupid. OK, it doesn’t actually say that in the book, but that’s what it’s getting at.
- The over-reliance on quantitative indicators for decision-making leads to data corruption and distortion of the educational process. Think Atlanta cheating scandal. And see CL #87, which is one of my favorites, on Campbell’s Law specifically. And funnily enough, I wrote that one at Ann’s house in Seattle, my home away from home, which is where I am now.
- The data that we frequently want people to pay attention to, because they matter to us, frequently come too late in the process for those data to be useful to those people. You should check out Stiggins & DuFour, 2009, which we summarize really briefly on p. 121: “At the institutional level, boards of education and superintendents need outcomes-based assessment data in order to make budget and policy decisions… At the program level, principals, department chairs, and teacher teams need information to tell them… whether these plans are leading to improvement for students. And at the classroom level, teachers and students need continuous information about the individual students’ academic progress and experience of schooling.” Don’t ask people to look at data that they don’t control.
- Data don’t have to be outcome measures, and they don’t have to be numbers at all. “We have learned the hard way that we should not ask the question, ‘What data do you have?’ without explaining first of all that we mean any evidence about the question under consideration; otherwise, we will be told that there are no data, only to find out later that actually all kinds of information are available…” My co-author Rydell along with my colleague Tisha are exploring research related to transformative student voice which centers the experiences of students at the margins and positions students as genuine partners focused on improvement, with a view to organizing a convening about student voice. Add yourself to this mailing list if you want to be kept informed.
Then there’s a whole section on data teams and what they should look like, but I think that’s a CL all by itself. We had a lot to say about those… And more about data in future CLs; I’ve been getting a lot out of reading How to Measure Anything…, which has given me a frame of reference to think about several aspects of improving education. Here I just want to mention one other point about data, that is expressed really forcefully in How to Measure Anything: data not collected for the purpose of making a decision are probably not worth collecting. For example, lots of districts have protocols for learning walks, and during those they collect a lot of really interesting and potentially useful data. But if there are no decisions to be made as a result of collecting the data, what is the point? I don’t think it’s enough to say the learning walks will provide information about how well [fill in the blank] is being implemented. Super. But what is the decision that you need data in order to make?
One last quick plug: Behavioral Scientist magazine publishes a summer reading list—if you are looking for some summer reading, this is a good resource. Several are on my radar: Cultures of Growth: How the New Science of Mindset Can Transform Individuals, Teams, and Organizations; Everything Is Predictable: How Bayesian Statistics Explain Our World; The Friction Project: How Smart Leaders Make the Right Things Easier and the Wrong Things Harder; May Contain Lies: How Stories, Statistics, and Studies Exploit Our Biases—And What We Can Do about It; The Ritual Effect: From Habit to Ritual, Harness the Surprising Power of Everyday Actions; What Works, What Doesn’t (and When): Case Studies in Applied Behavioral Science; Smart Management: How Simple Heuristics Help Leaders Make Good Decisions in an Uncertain World. That seems like a lot… But I do love how some of them appear to contradict each other.
Finally, I’m working remotely from Seattle this week because at the end of the week is my friend Ann’s retirement party. So here’s a small homage to her and my relationship with her. I met Ann in January 1991 (I know, right?) in a grad school class called Dyslexia and Corrective Reading (where, incidentally, I learned about the research that is currently, all these decades later, fueling the latest battle in the reading wars). She came up to me at the end of class one day and announced that we were going to be friends, which was the first time that anything like that had happened to me. Since then we have had parallel careers as special ed teacher, regular ed teacher, assistant principal, principal, higher ed, consulting, in neighboring districts to begin with, and then thousands of miles apart, and with Ann usually a couple of years ahead. She is a student-centered and equity-forward leader with a big heart and a moral core, and I have benefited enormously from knowing her. She and I have supported each other through some challenging times. Every time we talk I come away with pages of notes and new ways of thinking about things, which is why she may think she’s retiring, but I have other plans; from my perspective, all that’s happening is that she’ll have more time to spend talking to me about what I’m working on. Fair warning, Ann.
Take care of yourself, take care of your people, and let me know if I can help. Best, Isobel
Isobel Stevenson, PhD PCC
860-576-9410
@IsobelTX
Author of The Coaching Letter
Author with Jennie Weiner of The Strategy Playbook for Educational Leaders: Principles and Processes Routledge, 2021
Author with Sarah Woulfin & Kerry Lord of Making Coaching Matter: Leading Continuous Improvement in Schools Teachers College Press, 2023
