What Greiner found when he began studying legal services was a complex, decentralized, and woefully underfunded endeavor that was helping only a fraction of the people in need. Across the country, hundreds of organizations worked more or less separately, without sharing information in any systematic way about the kinds of people who were coming to them for help and what exactly they needed. Most importantly, Greiner noted, no one had ever done any rigorous tests to see how much good these organizations were actually doing.
When he first approached the Harvard Legal Aid Bureau, an organization staffed by law students, Greiner made his pitch this way: Given that the number of people who come to HLAB looking for help far exceeds the number they can offer representation, why not randomize the sorting process, create a control group, and try to learn something about the organization’s impact? Initially, Greiner proposed running the study using either domestic abuse cases or eviction cases.
The answer from HLAB’s leaders was a firm no. “We pretty quickly decided we weren’t willing to do that,” said faculty director David Grossman, explaining that the student volunteers simply had too much confidence that their sorting methods were working to jettison them in favor of a random number generator. These methods, which took into account not only the likelihood that HLAB could make a difference in a particular case, but also the urgency of the case and the number of people affected by its outcome, were based on years of experience and institutional knowledge. Given the high stakes in eviction and domestic abuse cases, Grossman said, “randomizing away” clients HLAB would take under normal circumstances felt deeply immoral.
HLAB’s reluctance pointed to one of the main hurdles Greiner is facing as he looks for more legal services providers to work with on experiments: For an organization to do what he’s asking involves an admission that what they do normally might be ineffective. “Given our experience, we feel like we can pick the cases where we can make a difference,” said Greater Boston Legal Services executive director Jacquelynne Bowman. “But if it’s randomized, then we may miss out on providing representation to someone who truly needs it.”
Convincing professionals to doubt their methods is never easy, said Greiner, but it might be especially hard in the world of legal services. “Is it surprising that not every single legal aid attorney possesses the kind of self-doubt that might cause her to desire to undergo a rigorous evaluation of her...activities?” he asked. Nevertheless, Greiner believes this kind of self-doubt is important, precisely because people in the trenches can’t always look objectively at what they’re doing. “I don’t deny that...experiential observation is worthwhile,” he said. “But my job is to question and test it, because it turns out that human beings sometimes stink at figuring out what works and what doesn’t from day-to-day observations.”
The Harvard Legal Aid Bureau ultimately agreed to work with Greiner when he proposed running a study on people seeking unemployment benefits. This was an area of their practice they were already unsure of, said Tim Visser, the group’s president, with low enough stakes that the moral issues were not prohibitive. According to Visser, the students saw potential in Greiner’s study. In theory, it promised to reveal patterns that would tell them what to look for in clients.
Greiner’s surprising results, which were later published in the Yale Law Journal, indicated that people who were offered legal help from HLAB were no more likely to prevail in their cases—and worse, had to wait longer for their cases to be resolved—than those who were not. “That right there was a revelation,” said Greiner. More than 90 percent of people offered HLAB’s help received it; among those turned away, at least 39 percent found representation elsewhere. The fact that both groups scored a win rate of around 70 percent meant that getting help from HLAB was not increasing people’s chances as much as had been assumed.
HLAB swiftly responded to the study, sending a letter to colleagues in which they explained their participation and pointed out what they regarded as flaws in Greiner’s methodology. The letter was meant in part to address the concerns of legal aid workers around the country who had begun to worry about the study’s political implications. “When I first read the HLAB study and when I first heard Jim speak, I thought, ‘Oh, no. This is going to provide all kinds of fodder for people who just hate lawyers, or people who just hate poor people, and it’s going to cause all kinds of problems for us,’” said Steven Eppler-Epstein, the executive director of Connecticut Legal Services. But after speaking to Greiner, Eppler-Epstein concluded that the professor had a real, long-range plan to improve the legal services world, not hurt it, and today he is one of Greiner’s closest allies.Continued...