News

Hard work pays off on the playing fields, as former University of North Texas student Miranda Holguin shows. A new study in Texas shows where schools pay off for students in the labor market. (AP Photo/Brandon Wade)Copyright 2020 The Associated Press. All rights reserved.When it comes to college, the national tide has shifted to focus on student outcomes and return on investment (ROI).

More states and the federal government are aligning on making sure that programs eligible for government dollars do no harm to the students who enroll.

With the release of a value-added earnings outcomes study in Texas that measured the economic outcomes of 935,767 students who enrolled in 86 public institutions in Texas between 2008–09 and 2018–19 and pursued a bachelor’s degree, associate’s degree, or certificate, the Postsecondary Commission, which accredits institutions that produce strong economic returns for their students, alongside Mathematica, have made meaningful progress in showing the field how to responsibly and credibly measure ROI (full disclosure: I am on the Postsecondary Commission’s advisory board). The release of the study also shows how much more work governments have to do to create the underlying infrastructure to make this kind of work possible everywhere in the country.

The broader move to focus on higher education outcomes has been a welcome development for a sector that doesn’t boast stellar results.

Nearly 4 in 10 students don’t graduate from four-year schools within six years and, according to research published in 2024 by the Foundation for Research on Equal Opportunity, nearly a quarter of bachelor’s degrees have a negative ROI; 43 percent of associate degrees have a negative ROI; and 43 percent of master’s degrees have a negative ROI.

And yet the move toward outcomes has remained incomplete.

How to measure and evaluate programs such that it doesn’t encourage colleges to “cream” students by admitting students who are already the furthest ahead in their abilities? How to measure such that the findings aren’t simply referendums on the local labor markets—or end up comparing the averages of a local labor market with those of a much bigger, more diverse state? How to get to as good a counterfactual as possible for any given student to help them understand whether enrolling is likely to have a positive economic outcome?

I’ve long had an interest in this area. I published the “Quality-Value Index” in a 2010 op-ed in the Washington Times, refined it further in a 2011 white paper for the Center for American Progress, and then founded a non-profit, the Education Quality Outcomes Standards board, in 2015 to offer a set of standards for how institutions should measure and report on a broad set of student outcomes.

MORE FOR YOUOne of the metrics in that report we titled “Earnings boost”—a measure to capture the change in earnings students experience from prior to attending their program to after. As we built the standard, we made a hand wave to the importance of designing the counterfactual for any given student, but didn’t know where to get the data or how to balance the various intricacies that would present themselves in reality.

That’s one of the most impressive things about the Postsecondary Commission’s Texas study. It sweats the details and then some to create something that is robust, reliable, and repeatable—as the Postsecondary Commission will be repeating the study and will broaden the sets of research questions on which it reports.

The most recent study measures the outcomes for all entrants—not just graduates. It factors in net cost to the student, including opportunity costs.

It constructs a matched-comparison group for each student to gauge the outcome. That matched-comparison group takes into account county of residence; prior earnings; standardized test scores from high school; age; family income; prior educational attainment; race/ethnicity; gender; and high school experience, including course-taking profile, attendance and disciplinary profile, English Language Learner status, and special education status.

It’s a dizzying array of attributes, made possible because of Texas’s impressive data infrastructure.

The study is consequently able to report out on outcomes by institution, program, and demographic.

To be clear, no observational methodology can perfectly recreate the counterfactual. Motivation, aspirations, and personal circumstances remain difficult to observe. What’s the Job to Be Done for why a student in a particular circumstance enrolled in a specific program versus another option? Three students with identical demographics and academic records may enroll in the same program for entirely different reasons. One may be attempting to get out of a dead-end job. Another may be at a stable point in their life seeking exploration and enrichment. A third may simply be trying to take the next step in their career. Those motivations can shape outcomes but are difficult to capture in administrative records.

Nevertheless, the Texas study represents one of the most sophisticated efforts yet to narrow that gap. And it seeks to account for some of these factors that are hard to measure by removing individuals who attended postsecondary education outside of Texas after high school, those who may have out-of-state earnings, and those under 21 who don’t have high school test-score data from the baseline samples.

The results from the Texas study aren’t necessarily shocking. They show that the college degree does typically pay off. If anything, the outcomes appear stronger than some previous ROI analyses have suggested. One possibility is that the study’s richer matching methodology produces a more precise estimate of institutional impact.

But there is lots of important nuance. Certain schools and programs have negative ROIs. Certain programs don’t pay off. Different areas of study promise different returns.

This isn’t to say that economic return is the only measure we should care about when it comes to a college program. There are many reasons to pursue an education.

But it is to say that we should set a standard of doing no economic harm to students. And this methodology helps us understand the underlying data streams required to put those robust guardrails in place.

Sadly, Texas is one of the few states that has this type of data. But with the publication of this report and, hopefully, more colleges choosing to be accredited by the Postsecondary Commission in future years, perhaps the next chapter in higher education outcomes will be written in laying the governmental infrastructure required to make sure that we don’t leave students worse off when they enroll in college—regardless of where they live.