In January, a team of researchers from Vanderbilt University released a study that seemed like a serious setback to the push for nationwide, universal pre-K programs. The study found that children who participated in a pre-kindergarten program in Tennessee in 2009 and 2010 had worse test scores and behavioral results as sixth-graders than children who did not. The study was released by pre-K critics as another blow to President Biden’s struggling Build Back Better bill, but the fallout was even greater. Media outlets and experts wondered: Is pre-K actually bad†
The Tennessee study was carefully designed and compared nearly 3,000 low-income children randomly selected from a group of applicants with a comparable control group that was not chosen. But pre-K isn’t bad, and the problem isn’t the study. It is how the language and techniques of academic research are mistranslated into how educational policy is understood by both the public and policy makers.
Pre-K has been offered in several states and municipalities for decades, yielding dozens of academic studies. Most have found positive effects on children† Less than a week after the Tennessee study was released, new research from Indiana found positive results for pre-K on test scores in grades three and four. As in Tennessee, the program serves low-income families. We’ve all learned to stay focused on poll averages when tracking political competitions, because even well-designed polls sometimes produce inaccurate results. Study results should be treated in the same way, and the study equivalent of the poll mean for pre-K – looking at multiple studies instead of just one – remains consistent and strong†
Nevertheless, negative results require attention. One explanation for the poor results is that pre-K education in Tennessee around 2009 and 2010 was not as good as it should have been. While childcare in itself is an important benefit for working parents, it is not enough, academically, to just open a room where small children can be all day. A good preschool has well-trained teachers who understand how to structure the environment to encourage the development of language and cognitive skills. This means no rigid instruction, but many well-designed opportunities for enrichment and play.
The study authors provide evidence that Tennessee’s program was comparable in quality to other states. But there is reason to believe that the overall quality was still not so good. Between 2009 and 2012, researchers, including two of the co-authors of the new study, evaluated a sample of 160 pre-K classrooms in Tennessee using a widely used research tool called the Early Childhood Environment Rating Scale (ECERS), which assesses the classroom design, environment, curricula, discipline, and strategies teachers use to promote language and literacy, evaluates. Only 15 percent of classrooms scored as “good” or higher. Eleven classrooms scored below the “minimal” quality.
The Tennessee legislature went on to pass the Pre-K Quality Act of 2016, which was designed to improve classroom curricula, provide training for pre-K teachers, and strengthen coordination with elementary schools.
While the negative results of the Tennessee study were highly unusual, the finding of diminishing returns was not. A number of other studies have shown that the academic benefits of pre-K sometimes fade over time. The authors speculate whether some pre-K classrooms may have focused too much on discrete, measurable goals like “knowing your ABCs,” at the expense of broader literacy and executive functioning skills that matter in later years. It’s a fair question.
But the Tennessee study and the resulting backlash also illustrate a broader problem, not limited to education, in how research methods define complex systems and how the media describes those results.
Research into what works is important, but it has its limits
The negative effects of pre-K in the Tennessee study were “statistically significant.” In normal language, “significant” means “substantial” or “non-trivial”. In statistics, “significant” means something else: “A difference that is most likely not random.”
Imagine you throw a gallon of white paint on a football field. Statistical significance means “the field is generally unequivocally less green and whiter than it was before you dumped the paint.” It does not mean that “a substantial part of the field is now white”. An effect can be simultaneously statistically significant and practically insignificant.
The Tennessee study found that children who attended pre-K had a 97.1 percent attendance rate in sixth grade, while children who did not attend pre-K had a 97.5 percent attendance rate; there were no significant differences in attendance in grades 1-5. This finding was reported in New York magazine because “pre-K participants were also significantly more likely to miss class.” What they were, in statistical terms – but what it ultimately refers to is a 0.4 percentage point difference in one year out of six. (The same comment can be made about some of the much larger number of positives pre-K results: They are statistically significant, but not particularly large.)
The much larger debate about whether pre-K is worth the kind of massive national investment proposed by President Biden is often ill-served by the vocabulary and practice of modern social science, especially the kind that lead to journal articles and promotions of tenures.
The Tennessee study uses powerful statistical techniques designed to find meaning in a fog of information. In a world full of personal stories, anecdotes and ideologies, these methods, which have been greatly refined and improved over time, are incredibly important. They help distinguish causation from correlation, pattern from chance, truth from fiction.
But they also impose a very specific mental model on everything they investigate. Studies aim to establish a discrete action and determine what happened next as a result of that event, and that event alone. They’re perfect for evaluating something we’re all too familiar with by now: vaccines. FDA studies randomly assign people to one of two groups. One gets the drug, the other a placebo. They wait a certain amount of time and see if the people who have received the drug are less sick. Not coincidentally, the authors of the Tennessee study describe pre-K as a “treatment,” standard language in the social sciences.
The problem is that pre-K doesn’t look very much like a vaccine. Raising a child is more like building a house. Nobody sees walls, windows and roofs as discreet interventions intended to keep people warm and dry. They are parts of a greater whole. If the roof leaks, you will get wet. If the windows break, you will get wet. Cracked foundation? Wet. All parts must work together at the same time.
Many early childhood education initiatives, such as Head Start and the Tennessee program, have been given to children living in impoverished, sometimes traumatic environments. The public schools in their neighborhood are often underfunded and underperforming. Work and care are scarce. Giving them pre-K can be like helping an out-of-home person by building a single wall on a vacant lot. One wall is better than no walls, but they are still exposed to the elements above and on three sides.
Some of the most effective early learning programs provided a range of social, parental and health support beyond education. (The Biden plan, which includes early childcare funding, a child tax credit, improved health care coverage, community college, and so on, follows a similar path.) Some of the least effective were implicitly based on the hope that an extra year of school could help children. inoculate against the risk of academic failure, saving policymakers the effort and expense of improving the next 13 degrees. If the consistent initial benefits of early childhood education sometimes fade, we should focus on the schools and classes where the gains are diminishing.
The distinction between components and structure helps to explain a long-standing conundrum in educational research. At the level of nations, populations and individuals, the benefits of education are enormous. Highly educated people do better on almost every economic and social measure: income, health, longevity, and so on.
But researchers have so far struggled to isolate the effects of specific parts of the educational whole. It is absolutely impossible to write an article like “Effects of a Statewide Pre-Kindergarten Program on the Performance and Behavior of Sixth Grade Children” and publish it in a peer-reviewed academic journal without at least 20 years of formal education. But the statistical techniques you pick up along the way cannot explain exactly why.
The pre-K debate is also subject to some bigger misconceptions. Noah Smith, an economist and popular blogger, offered: an honest and thorough summary of the pre-K research in his Substack newsletter, in which he concluded that while pre-K may provide more benefits to underprivileged children who do not have an enriching, stable home environment, “there are many children who are likely to be injured by forcing them into universal pre-K- programs.” But universal pre-K programs are not required; in the vast majority of states, even kindergarten is not required. In Tennessee, only 22 percent of the state’s 4-year-olds are enrolled.
There is an important place for research such as the Tennessee study in the conduct of education policy. It can help teachers understand what works best and how they can improve.
But for more existential questions — like whether universal pre-K should even exist in the US at all — it’s helpful to start with what privileged people give their own children. The Capitol Hill neighborhood of Washington, DC, for example, is home to many of the highly skilled staffers, lobbyists, and lawmakers who will help determine the ultimate fate of Build Back Better. Where do their 3- and 4-year-old children receive education? Many are in pre-K in non-tuition public schools. Sometimes counterintuitive research results are like that for a reason.