they failed to continue improving with treatment. (The authors measured improvement via self-reports in answer to questions such as “Have you been sad the past month?” or “Have you participated in social activities?”) What their conclusion doesn’t address, however, is the possibility that the people in treatment were
already
doing better than the AA group, and that they therefore had less room to improve over those last eight years. We do not know, nor do the researchers say, which interpretation is right.
More problematic is that the study elided some potentially telling fluctuations in the data. People who stayed in AA for fewer than six months had w
orse
outcomes than people who never entered AA at all. This finding seems to mirror the Brandsma data: AA attendees seem to get worse before they get better. One theory is that the finding is nothing but
noise
—the standard statistical turbulence that can foul any short-term study. But if the data are real and repeatable, then they suggest something the Moos researchers perhaps did not consider: that AA might do more harm than good for the people who
choose
to attend but do not
buy into
the program.
The Moos study also employs some objectionable statistical methods. In one critical omission, its conclusions ignore all the people who died and the large number of people who dropped out of the study altogether, despite conceding that these were the people who statistically consumed the most alcohol. As early as year eight, the number of subjects who were left in AA had already shrunk by nearly 40 percent (from 269 to 166), yet these people are erased from the conclusions as if they had never existed at all. Add up all the people who died and the dropouts, and the results for AA become far grimmer than the authors suggest.
The stated size of this survey is also misleading. Although the researchers began with 628 people, the total number of people who remained through the sixteen-year follow-up
and
also stayed in AA for longer than six months—that is, the group on which the authors’ major findings are based—was just 107, or 17 percent of the original sample. And of the remaining 107, the researchers never revealed the actual number of people who improved, or even stayed sober. They told us only which group “had better outcomes.”
Next, there is the question of validity of the results. As I have mentioned, self-reporting is a tricky methodology, prone to the illusions of self-deception and imperfect memory. In most observational research, surveys are the standard currency—without surveys, there can be no data. Yet there are ways to mitigate the information people report about themselves, notably independent testing. The Moos study did not attempt to independently verify any of the surveys it was based on. (The Fiorentine group, by contrast, supplemented their surveys with urine tests.)
Finally, the punctuated nature of the study addressed only the six-month windows prior to each of the four check-ins. This meant that of the sixteen years covered by the study, the researchers’ surveys gathered information on only two of them. No questions were ever asked about the stretches of time in between follow-ups; 88 percent of the time was never studied. As the authors acknowledged in the 2006 paper, “Another limitation is that we obtained information only on 6-month windows of alcohol-related outcomes at each follow-up, and thus cannot trace the complete drinking status of respondents over the 16-year interval.”
Ultimately none of these issues should be great enough to disqualify the Moos study on its own. But together they should give us pause. The study had no controls, so subjects were free to join and leave treatment as they wished. And for every slice of subjects that got better, the study omitted many about which we are never told a thing. Possibly as a consequence of these limitations, the authors of the study readily acknowledged that they, too,
already
doing better than the AA group, and that they therefore had less room to improve over those last eight years. We do not know, nor do the researchers say, which interpretation is right.
More problematic is that the study elided some potentially telling fluctuations in the data. People who stayed in AA for fewer than six months had w
orse
outcomes than people who never entered AA at all. This finding seems to mirror the Brandsma data: AA attendees seem to get worse before they get better. One theory is that the finding is nothing but
noise
—the standard statistical turbulence that can foul any short-term study. But if the data are real and repeatable, then they suggest something the Moos researchers perhaps did not consider: that AA might do more harm than good for the people who
choose
to attend but do not
buy into
the program.
The Moos study also employs some objectionable statistical methods. In one critical omission, its conclusions ignore all the people who died and the large number of people who dropped out of the study altogether, despite conceding that these were the people who statistically consumed the most alcohol. As early as year eight, the number of subjects who were left in AA had already shrunk by nearly 40 percent (from 269 to 166), yet these people are erased from the conclusions as if they had never existed at all. Add up all the people who died and the dropouts, and the results for AA become far grimmer than the authors suggest.
The stated size of this survey is also misleading. Although the researchers began with 628 people, the total number of people who remained through the sixteen-year follow-up
and
also stayed in AA for longer than six months—that is, the group on which the authors’ major findings are based—was just 107, or 17 percent of the original sample. And of the remaining 107, the researchers never revealed the actual number of people who improved, or even stayed sober. They told us only which group “had better outcomes.”
Next, there is the question of validity of the results. As I have mentioned, self-reporting is a tricky methodology, prone to the illusions of self-deception and imperfect memory. In most observational research, surveys are the standard currency—without surveys, there can be no data. Yet there are ways to mitigate the information people report about themselves, notably independent testing. The Moos study did not attempt to independently verify any of the surveys it was based on. (The Fiorentine group, by contrast, supplemented their surveys with urine tests.)
Finally, the punctuated nature of the study addressed only the six-month windows prior to each of the four check-ins. This meant that of the sixteen years covered by the study, the researchers’ surveys gathered information on only two of them. No questions were ever asked about the stretches of time in between follow-ups; 88 percent of the time was never studied. As the authors acknowledged in the 2006 paper, “Another limitation is that we obtained information only on 6-month windows of alcohol-related outcomes at each follow-up, and thus cannot trace the complete drinking status of respondents over the 16-year interval.”
Ultimately none of these issues should be great enough to disqualify the Moos study on its own. But together they should give us pause. The study had no controls, so subjects were free to join and leave treatment as they wished. And for every slice of subjects that got better, the study omitted many about which we are never told a thing. Possibly as a consequence of these limitations, the authors of the study readily acknowledged that they, too,
Similar Books
