Researchers find AI is bad at predicting GPA, grit, eviction, job training, layoffs, and material hardship

A paper coauthored by over 112 researchers across 160 data and social science teams found that AI and statistical models, when used to predict six life outcomes for children, parents, and households, weren’t very accurate even when trained on 13,000 data points from over 4,000 families. They assert that the work is a cautionary tale…

Researchers find AI is bad at predicting GPA, grit, eviction, job training, layoffs, and material hardship

A paper coauthored by over 112 researchers across 160 data and social science teams found that AI and statistical models, when used to predict six life outcomes for children, parents, and households, weren’t very accurate even when trained on 13,000 data points from over 4,000 families. They assert that the work is a cautionary tale on the use of predictive modeling, especially in the criminal justice system and social support programs.
“Here’s a setting where we have hundreds of participants and a rich data set, and even the best AI results are still not accurate,” said study co-lead author Matt Salganik, a professor of sociology at Princeton and interim director of the Center for Information Technology Policy at the Woodrow Wilson School of Public and International Affairs. “These results show us that machine learning isn’t magic; there are clearly other factors at play when it comes to predicting the life course.”
Fragile Families Study
The study, which was published this week in the journal Proceedings of the National Academy of Sciences, is the fruit of the Fragile Families Challenge, a multi-year collaboration that sought to recruit researchers to complete a predictive task by predicting the same outcomes using the same data. Over 457 groups applied, of which 160 were selected to participate, and their predictions were evaluated with an error metric that assessed their ability to predict held-out data (i.e., data held by the organizer and not available to the participants).
The Challenge was an outgrowth of the Fragile Families Study (formerly Fragile Families and Child Wellbeing Study) based at Princeton, Columbia University, and the University of Michigan, which has been studying a cohort of about 5,000 children born in 20 large American cities between 1998 and 2000. It’s designed to oversample births to unmarried couples in those cities, and to address four questions of interest to researchers and policymakers:
The conditions and capabilities of unmarried parents
The nature of the relationships between unmarried parents
How the children born into these families fare
How policies and environmental conditions affect families and children
“When we began, I really didn’t know what a mass collaboration was, but I knew it would be a good idea to introduce our data to a new group of researchers: data scientists,” said Sara McLanahan, the William S. Tod Professor of Sociology and Public Affairs at Princeton. “The results were eye-opening.”
The Fragile Families Study data set consists of modules, each of which is made up of roughly 10 sections, where each section includes questions about a topic asked of the children’s parents, caregivers, teachers, and the children themselves. For example, a mother who recently gave birth might be asked about relationships with extended k
Read More From Publisher