Analyzing whether new policies and practices to reform the criminal justice system actually work can be time-consuming and costly. But one professor is rethinking the way such evaluations are done.
The latest episode of Matt Watkins’s podcast, New Thinking, for the Center for Court Innovation, features a professor who has applied a strategy used successfully by Amazon CEO Jeff Bezos to test multiple ideas with minimum cost.
Angela Hawken, a professor of public policy at New York University, is the founder of BetaGov, a program that offers free and fast evaluations of public policy programs.
Hawken says that too many research projects use the “Cadillac” model, employing expensive, time-consuming tests that make it difficult for researchers to halt a study if its results aren’t productive.
In developing her model, Hawken looked to successful companies like Amazon. She cites Bezos as saying that he attributes the success of Amazon to its ability to run thousands of low-costs trials of ideas or apps at one time to test whether they attract an audience.
Hawken set out to make what she calls an alternative, “exploratory” track of research that is “inexpensive to do and can be shut down very nimbly if the outcomes aren’t moving in the intended direction.”
BetaGov uses mostly randomized controlled trials, or RCT’s. Hawken says RCT’s are the “gold standard” of research because they verify causal claims, which determine the effect of the intervention being measured between two similar populations.
RCT’s are especially valuable in criminal justice research, which Hawken says frequently suffers from selection biases that can skew results.
“You’ll see many evaluations in criminal justice that compare, for example, treatment completers to people who did not go to treatment,” said Hawken. But the results of this comparison are misleading, because “the sorts of people who complete anything are different from the sorts of people who either don’t start or drop out along the way.”
Another major innovation of BetaGov is the use of practitioners on the front lines of public policy and justice systems across the country in place of a large research team. Hawken refers to them as “pracademics”–people who work in government organizations and want to test ideas that might be of value.
Hawken describes them as those in the public sector “who have looked around their offices and their buildings and the clients that they were serving and asked themselves the what-if question. What if we did it differently?”
If a practitioner runs a test and finds promising outcomes, it’s quickly replicated in another location to see if it holds up.
One benefit of this democratized approach is that it allows for more diverse input on what gets researched.
“Very few people typically get to weigh in on what will become an evidence-based program or practice,” Hawken said. “We wanted to shatter the monopoly over who gets to decide what is being tested.”
This approach extends further to what Hawken calls “collaborative design”— listening to ideas from people who are closest to the problems facing the criminal justice system.
Hawken spoke about her time in a Washington state meeting inmates in solitary confinement. Their ideas were so useful that Hawken says they turned into policy reforms in several states.
Hawken cited a paper written by a roboticist at MIT in the 1980s as another source of inspiration for BetaGov. The paper argued that instead of sending massive probes or satellites into space, researchers should send thousands of little “bugs,” that would signal when they detected something valuable.
“And I thought, wow, what if we did that in the government sector?” she said. “We blasted out thousands of innovations and then let the promising ones cluster.”
“We tend to send out the big mission first, then find it’s dark, and then get mad,” said Hawken. “We need to change how we do this.”
“That doesn’t mean there isn’t room for the traditional academic model, of course there is. But it’s a parallel track.”
Accepting that most attempts will fail is central to the approach behind BetaGov as well. Its low-cost structure helps to engender this line of thinking.
“When no money’s at stake, people are also more willing to fail,” Hawken said. That’s so important, because there’s failure all around us.”
Dane Stallone is a TCR news intern. Readers’ comments are welcome.