The Triumph of Bad Science
If we want to understand how our society may end up deluded about the merits of psychiatric medications, we can look at the research published by Robert Gibbons, Director of the Center for Health Statistics at the University of Chicago, on antidepressants and their use in children and adolescents. His latest articles appear in the June issue of the Archives of General Psychiatry, and if we examine his research, and look at how critiques of his research have been treated, we can see how bad science ends up creating a false “evidence base” for the use of the medications.
Let’s follow this story from its start.
In 2004, the FDA concluded that in randomized trials, SSRI antidepressants doubled the risk of suicidal thoughts and behaviors in children and young adults, compared to placebo. That finding led the FDA to issue a “black-box warning” that these drugs could increase the risk of suicide in children and adolescents.
Gibbons was a member of the FDA panel that voted in favor of the black box warning, fifteen to eight. However, he was one of the dissenting eight, and, as he recently recalled in an interview, he felt that the warning was not warranted. Ever since then, he has published a number of articles that dispute the FDA’s finding that SSRIs increase the risk of suicidal thoughts and behaviors.
As his most recent articles disclose, he has also served as an expert witness for Wyeth and Pfizer Pharmaceuticals in cases related to antidepressants and suicide. His findings, it is fair to say, help make him a valuable witness for the makers of SSRIs.
One of his first such papers, which attracted a great deal of media attention, was published in the American Journal of Psychiatry in 2007. He reported that in the wake of the black box warning (and a similar warning by European regulatory authorities), the prescribing of SSRIs to children and adolescents decreased in the U.S. and Europe, and that when this happened, there was a dramatic increase in suicides in the two countries he studied, the U.S. and the Netherlands. The black box warnings, he concluded, apparently led to an increase in pediatric suicides.
Critics quickly pointed out the dishonest science that Gibbons had employed to make this case. He reported that SSRI prescriptions to youth declined by 22% in the U.S. from 2003 to 2005, and that suicide rates in youth rose 14% between 2003 and 2004. But since he had only the suicide rates for the U.S. through 2004, he should have focused on prescribing rates during that same period of time.
In fact, there had only been a very small decrease in the prescribing of SSRIs to youth between 2003 and 2004, when the number of suicides rose. It was between 2004 and 2005 that the there was a significant decrease in the prescribing of SSRIs to youth, and–as the critics noted–once the suicide data for that period became available, it showed that during that time, the number of suicides for persons ages 5 to 24 declined.
In other words, the data showed that as the number of prescriptions to children and youth declined, the number of suicides in this age group declined too. But Gibbons reported that the opposite was true. He did so by matching the increase in suicides in 2003-2004 to the decline in prescribing in 2004-2005. This is not the sort of error a scientist “accidentally makes.” This is the sort of presentation of data one makes when he or she is trying to deliberately tell a story that fits a preconceived end.
In the Netherlands, Dutch academics were incensed with Gibbons and his statistical antics. In the Dutch Drug Bulletin, they noted that the increase in suicides in the Netherlands was so small that it was “not statistically significant.” They described his conclusions as “astonishing” and “misleading,” and stated that Gibbons and his co-authors had been “reckless” to publish such claims.
But how did the U.S. media treat Gibbons’ article? Newspapers took his conclusion at face value. Gibbons, the Chicago Tribune reported, had “documented a close correlation between declining use of the antidepressants known as SSRIs and rising suicide rates among young people up to age 19.” And given that “fact,” Gibbons told the paper that the FDA’s black box warning had a “horrible and unintended effect” and should be withdrawn.
All told, Gibbons published at least eight papers between 2005 and 2011 challenging the FDA’s black box warning (and it would be possible to critique those papers as well.) Then, in February and March of this year, the Archives of General Psychiatry published in its online edition two more of Gibbons’ articles on this topic. These two articles have now appeared in the journal’s June print edition.
In one article, “Suicidal Thoughts and Behavior With Antidepressant Treatment,” Gibbons reported that he had done a “reanalysis” of the randomized placebo-controlled studies of fluoxetine and venlafaxine, and found “no evidence of increased suicide risk in youths receiving active medication.” In the second article, “Benefits from Antidepressants,” he reported that these drugs were highly effective in reducing depressive symptoms in youths too.
His reports, starting with their online publication, attracted considerable media attention. In an interview with the LA Times, Gibbons again sounded the theme that the black box warning issued in 2004 was a mistake. “The greatest cause of suicide is untreated or undiagnosed depression,” he said. “It’s very important that this condition be recognized and appropriately treated and not discarded because doctors are afraid to be sued.”
On NPR, it was more of the same. When the FDA issued its black box warning, Gibbons said, “I worried that what we might end up with was a real epidemic of suicide. And the data suggests that this is exactly what happened. Rather than the black-box warnings leading to decreases in child suicide rates, they were followed by some of the largest increases in child suicide rates both here in America and around the world.”
More recently, with the publication of the two articles in the June print edition of the Archives of General Psychiatry, Medscape reported that his findings indicated there might be a “need to revaluate” the black box warning.
Back in February, when Gibbons’ “Suicidal Thoughts” article first appeared online, David Healy wrote a blog detailing—as he said—the many “tricks” that Gibbons had employed to make the case that fluoxetine and velafaxine didn’t increase suicidal thoughts in youth. In a similar vein, Mickey Nardo, a retired psychoanalyst who writes the blog 1boringoldman, wrote a series of posts on the two articles, describing the “inappropriate data selection,” “opaque methodology,” “obvious arithmetic errors” and “deceitful presentation” to be found in the two studies. Such flaws, he noted, rendered the studies incapable of “supporting any broad conclusions about the safety or efficacy of antidepressants in youth.”
Intent on making his criticisms part of the scientific discussion, Nardo sent a “letter to the editor” of the Archives of General Psychiatry with these criticisms. He did so with the expectation that his letter would be published in the print journal. If so, his criticism would then become part of the scientific record that is archived by PubMed.
The AGP editors neatly kept that from happening. They decided to publish his criticism as a “readers reply,” and only online. As a result, they informed Nardo, “like other online posts, your reply will not be indexed in PubMed.”
As such, his criticism of Gibbons’ report is not part of the “evidence base” on this topic.
More recently, Matthew Miller, an associate professor at the Harvard School of Public Health, whose research focuses on suicide, took a critical look at Gibbons’ newly published suicide article. (Disclosure: Matthew Miller is a friend of mine.) He and his colleagues who collaborated on this review also concluded that Gibbons’ finding—that fluoxetine and venlafaxine didn’t increase the suicide risk in youth—was unwarranted. Instead, as they detailed in a letter to the editor of the Archives of General Psychiatry, Gibbons had committed methodological errors and misinterpreted data to draw “misleading conclusions.” In fact, Miller and his colleagues concluded that the very data that Gibbons presented in his study, when properly analyzed, “align with the FDA findings,” which is that the effect of antidepressants on suicidality in youth “appears harmful.”
The AGP editors treated their “letter to the editor” in the same way they had treated Nardo’s submission. They published it as a “reader’s reply,” but not as a letter to the editor. Once again, in this way, they kept this criticism from being archived in PubMed, and thus part of the searchable record.
In response, Miller wrote to the journal’s editor, Joseph Coyle, urging that their criticism appear in the print edition. Coyle didn’t respond.
Miller also sent an email to Medscape, protesting its touting of the Gibbons’ article as evidence that the two antidepressants didn’t increase suicidal behavior in youth. Medscape didn’t respond to him either.
In this brief review of Gibbons’ work—and a review of how critiques of his work were handled by the Archives of General Psychiatry–we can see the triumph of bad science. In randomized trials, SSRIs were shown to double the risk of suicidal thoughts and behavior in children and adolescents. This led the FDA to issue its black box warning. But since then Robert Gibbons has sought to tell a different story, both in the medical journals and in the media, and he has succeeded in doing so.
Critics may have revealed the bad science involved in his reports, but that criticism doesn’t substantially affect the bottom-line conclusion that shows up in PubMed and in the media: Researchers have found that SSRIs do not increase the risk of suicidal thoughts and behaviors in children and adolescents, and the real tragedy is that an unwarranted black box warning may be keeping some depressed youth from getting the drug treatment they need.
As such, this story can help us understand why we, as a society, may end up deluded about the merits of psychiatric medications. The evidence base is massaged in a way that protects the image of the drugs. Dishonest science gets published in the Archives of General Psychiatry and is archived in PubMed, while in-depth criticisms of that bad science are relegated to the “readers’ reply” corner of the journal’s online website, and thus excluded from the PubMed archives. Meanwhile, the media tells of Gibbons’ “findings,” but omits the part about the scientific dishonesty at the heart of those reports.
And voila, you have a process for creating a societal delusion.