Hello. The previous lesson separated eligibility, an award, and actual enrollment: each measures a different stage of access. The same distinction matters when a study claims that school choice improved an outcome. A family choosing a school, a child being offered funding, and a child using that funding are not interchangeable events.
This lesson gives you a way to assess a causal claim without advanced statistics. You will examine how children entered a study, what made its comparison group credible, and whether outcomes were missing at follow-up. A published review of a New York voucher study will provide a concrete example. It is evidence about that study, not a finding about Texas Education Freedom Accounts (TEFA).
The comparison a causal claim needs
Suppose children at participating private schools have higher average test scores than children at nearby public schools. The difference is real as a description of those groups. It does not, on its own, show that attending the private schools produced the difference. Children may have differed before enrollment, and families may have chosen schools partly because of those differences.
A causal study tries to approximate an impossible comparison: the outcome the same child would have had with and without the school-choice opportunity. Since only one of those outcomes can be observed, researchers need another group that offers a credible picture of what would otherwise have happened.
RAND’s discussion of alternative-school research is a useful starting point. It discusses charter and magnet schools rather than TEFA, but the problem it identifies—families and schools selecting one another—also applies when interpreting voucher studies.
Navigating Research on Alternative Schools | RAND
Read RAND’s explanation of why higher scores among students at a chosen school do not automatically establish that the school caused them.
In the opening passage beginning “Question what's responsible for good student performance,” read the selection discussion. Then find the paragraph beginning “Proving that a charter or other alternative school is responsible for higher achievement” and read the comparison discussion. Focus on what a comparison group must account for, including differences that researchers cannot easily measure.
Tracking each child’s progress over time is more informative than comparing scores at a single point. Yet improvement alone does not settle causation: children at another school might also have improved. Researchers therefore look for a comparison group whose members had a similar prospect of improving before the school-choice opportunity arose.
Matching children on observed characteristics, such as prior scores, can help. It cannot necessarily match family motivation, support, or other unmeasured factors related to both school choice and later outcomes. That is why the method used to form the groups matters more than whether their eventual averages look different.
Selection into a study is different from assignment within it
A lottery can make a comparison particularly credible if eligible participants are assigned by chance to receive an offer or not. Among the people in that lottery, random assignment tends to distribute both measured and unmeasured starting characteristics across the groups. It does not guarantee perfect balance in every individual study, but it removes the deliberate sorting that occurs when researchers simply compare families who chose different schools.
This short video explains the logic of random assignment. Its example comes from health research; for school choice, think of the “intervention” as an offer of funding, not a pill or a guarantee of private-school attendance.
Randomized Control Trials and Confounding
Watch “Randomized Control Trials and Confounding” from Global Health with Greg Martin for a visual explanation of why an alternative cause can distort a comparison and how random assignment helps.
Watch random assignment for the distinction between joining a study and being assigned to a group. Then watch confounding for the reason random assignment can help with factors researchers have not measured. Treat the video’s description of groups as “identical” as the goal of randomization, not a promise that every characteristic will be exactly equal.
There are two selection questions, and answering one does not answer the other. Who entered the study? If only interested, eligible applicants entered a lottery, its result does not automatically describe all children. How were those applicants divided? A genuinely random division can still support a strong causal claim for the group studied. The boundary matters when a headline turns a finding about applicants into a claim about every Texas household.
A voucher study with a credible comparison
The U.S. Department of Education’s What Works Clearinghouse (WWC) reviewed a study of a privately funded New York City voucher program. The review, dated May 2013, describes a study that began with applicants in 1997 and measured college enrollment through 2011. Its sequence lets us inspect both selection questions.
WWC Review of the Report "The Effects of School Vouchers on ...
Read the WWC review to distinguish initial interest in a voucher from random assignment to receive an offer, and to identify the outcome the researchers measured.
On p. 1, read “What is this study about?” and the program description under “Features of the School Choice Scholarships Foundation Program (SCSF).” Then read the findings under “What did the study find?” On p. 2, in “Appendix A: Study details,” find the “Study sample” subsection. Read the selection account, following both lotteries and noting the numbers assigned to each group.
More than 20,000 children initially expressed interest, but they were not all randomly assigned to receive or miss out on a voucher. The review describes a first lottery that selected applicants to be invited to an eligibility-verification session, followed by a lottery assigning students to the voucher-offer or comparison group. In all, 2,666 students were randomly assigned: 1,374 to receive a voucher offer and 1,292 to the comparison group.
For a causal reading, preserve the distinction between offer and use. The review reports that 78% of the offer group used a voucher at some point during the first three years. Comparing only users with non-users would discard the protection of random assignment: deciding whether to use an offered voucher may itself reflect family circumstances. Comparing children by their original assigned groups instead estimates the effect of being offered this program, including the fact that some families did not use it.
The WWC review rates the study as meeting its evidence standards without reservations. It reports no impact on college enrollment for the study sample as a whole, while reporting positive findings for the African-American subgroup on specified college-enrollment measures. Neither “vouchers raised college enrollment for everyone” nor “no child could benefit” accurately states those findings. The causal claim supported by the design concerns an offer to the studied applicants and the outcomes measured—not TEFA, not every voucher design, and not the isolated effect of attending a particular private school.
The comparison group and the children missing at follow-up
A comparison group is not necessarily a group that never entered private school. In the New York study, children assigned not to receive the initial voucher offer could make other choices. What protects the comparison is how the offer was assigned, not a rule that the two groups must have entirely different school experiences.
Follow-up introduces a separate risk. Even well-matched groups at the start may become misleading if researchers obtain outcomes for different kinds of children in each group. The WWC calls the absence of an outcome for an initially assigned participant attrition. The relevant checks are how many outcomes are missing overall, whether missingness differs between groups, and whether the missing children might have had different outcomes.
WWC Review of the Report "The Effects of School Vouchers on ...
Return to the WWC review for what happened to the comparison group and how completely the researchers observed the college-enrollment outcome.
On p. 3, read the “Comparison group” subsection, from its opening through the sentence about private-school attendance without a voucher. Continue into “Outcomes and measurement,” starting with “The primary outcome of interest,” to identify the college-enrollment measure and its data source. On p. 8, under “Glossary of Terms,” read the attrition definition.
College-enrollment data were available for 1,363 of 1,374 children assigned a voucher offer and 1,279 of 1,292 assigned to the comparison group. Thus, outcomes were missing for 11 and 13 children respectively—24 of the 2,666 randomly assigned. The losses were small and similar across groups. That supports confidence in this comparison more than a study with many missing outcomes would, although the counts alone cannot tell us what happened to each missing child.
The danger is easiest to see with deliberately exaggerated, illustrative numbers. Imagine a study that assigns 100 children to each group but follows only 80 in each. Among those followed, 48 in the offer group and 40 in the comparison group reach an outcome. The observed rates favor the offer group: versus . Yet if none of the 20 missing offer-group children reached the outcome and all 20 missing comparison-group children did, the rates for everyone originally assigned would instead be and . Equal numbers missing from each group did not prevent a reversal. This is not a description of the New York study; it shows why a follow-up count belongs beside a headline result.
A different problem is sometimes mistaken for attrition. A child who was offered a voucher but did not use it can still have a measured college outcome. Non-use changes what the offer accomplishes; a missing outcome changes what the researchers can observe.
Reading the next school-choice claim
When a report or news story says a program “caused” an outcome, a compact audit has three parts:
- Selection and assignment: Identify the children who could enter the study, then determine whether an offer was assigned by lottery, families selected their own schools, or researchers matched existing groups.
- The actual comparison: Identify who remained in each group and what differed between them. An offer-group comparison supports a claim about the offer; a comparison of self-selected attendees may not establish an effect of attendance.
- Follow-up: Find the original group sizes and the numbers with measured outcomes. Note substantial or uneven losses and any reason missing outcomes might be related to success or failure.
Applied to TEFA, this framework prevents an important shortcut. A participation count or a comparison of enrolled private-school children with public-school children is not, by itself, an impact study. Even where an award is allocated by lottery, a credible causal analysis would have to compare children assigned within the relevant lottery pool, keep track of offer versus use, and measure outcomes for both groups. Priority categories are not interchangeable comparison groups merely because a lottery operates somewhere in the program.
Conclusion
The strongest causal reading begins with how groups were formed, not with the size of their eventual difference. Random assignment among applicants can provide a credible comparison for the effect of an offer, while self-selection into schools can leave alternative explanations. Missing follow-up outcomes can weaken even an initially strong comparison; non-use of an offer is a distinct issue.
The next lesson will examine how far findings on achievement, longer-term outcomes, satisfaction, and public-school spillovers can inform expectations for Texas. A study can answer its own causal question well while still leaving open whether its result transfers to a different program and population.