assessing quality in scientific research and reporting

Below is an abridged version of a paper I submitted on 15 Sept. 2012 as part of my LIBR 501 class at UBC.

Problem! “Almighty Echo Chamber for Lies and Falsehoods”

Dr. Richard Cox observes that “For many, the computer and the resultant Information Age heralds a time when every person, with a modicum of cost, effort, and education, can harness more information in practical ways than ever before.” Nevertheless, such a wealth of information can lead to problems. Science journalist Jim Giles argues that “as well as acting as an almighty echo chamber for lies and falsehoods, the internet has given a more powerful voice to those who wish to sow confusion and conspiracy” (44). And even when there is no intent to mislead, poor reasoning, flawed research methods, and sloppy journalism contribute to the confusion.

Examples abound. In debates on important subjects such as autism and vaccines, climate change, the efficacy of homeopathic medicine, or, in an example from Giles, whether Obama’s healthcare reform will bring in so‐called “death panels” (46), it is difficult, in this internet echo chamber, to know which side to believe. Faced with conflicting data, internet users can become frustrated and misinformed at best, and risk actual physical danger at worst. Particularly in the area of scientific journalism, misinformation can be a serious problem.

Information professionals must be aware of the colliding perspectives on the internet and the media’s inclination to proliferate inaccuracies. At the same time, we should be aware of the grassroots efforts by bloggers and the like to address these issues, and how various new software tools are making this work easier and more inclusive.

Solution? “The Internet: Peer Reviewed”

Some promising software tools are starting to appear, and their arrival is of great interest not only to information professionals but also, indeed, to any consumer of content on the internet. “There’s a way to cut through the piles of nonsense on the internet,” says Giles in a recent article in The New Scientist (44). In “Truth Goggles,” he reports on various products that are, or soon will be, available to help readers separate truth from falsehoods. Among these is, open‐source software that will allow users to annotate anything found online without fear that the content owner can revise or remove the comment (46). inventor Dan Whaley calls it “the Internet, peer reviewed” (qtd. in Giles 46).

A key piece of the software is its ranking system, whereby users can rate each other’s contributions and, therefore, their credibility (Giles, 46). Just as you can assess the reputation of an eBay seller by checking how other customers have assessed her, read praise for or complaints about a pseudonymous Wikipedia editor by checking his talk page, and easily find the top‐ranked stories on Reddit, you will be able to see how peers rate a given contributor. This rating, in conjunction with other factors such as her productivity and how she has been rated by randomly‐selected moderators, helps rank the quality of her contributions (Giles, 46).

Bad Science Bad

A big reason why we need tools like is that the media are not sufficiently interested in bringing us the truth. This is not just an issue for gossip columns, which most of us already read with a healthy dose of scepticism; even so‐called science journalists are failing us. Science news with a sensationalistic spin gets attention, regardless of whether the journalist has properly read and understood the research, and regardless of the quality of the research itself. Ben Goldacre, who maintains the blog Bad Science, wrote a piece for The Guardian on the subject of the media’s role in spreading misinformation about the supposed link between the MMR vaccine and autism in children. He examined the take‐up of researcher Andrew Wakefield’s controversial findings:

Wakefield was at the centre of a media storm about the MMR vaccine and is now being blamed by journalists as if he were the only one at fault. In reality, the media are equally guilty. Even if it had been immaculately well conducted—and it certainly wasn’t—Wakefield’s “case series report” of 12 children’s clinical anecdotes would never have justified the conclusion that MMR causes autism, despite what journalists claimed: it simply didn’t have big enough numbers to do so.

But the media repeatedly reported the concerns of this one man, generally without giving methodological details of the research, either because they found it too complicated, inexplicably, or because to do so would have undermined their story. (“The Media Are Equally Guilty” 17)

In another article, this one for Significance, Goldacre describes how a study that found no statistically significant increase in cocaine use in children was misinterpreted and misrepresented in the popular press to such an extent that by the time it got to The New York Times, the headline read, “Cocaine Floods the Playground” (“When the Facts Get in the Way of a Story” 84).

No wonder people are confused.

Open Science Good and similar tools arrive at a time when we are already seeing an increased interest in openness, bottom‐up investigation, and the groomed assessments of the populace, thanks in large part to the wide adoption of social media. In the same way that open‐source software makes computer code available for anyone to review, use, and build upon for non‐commercial purposes, open science advocates want to make scientific research data available for examination by anyone, for the sake of the public good: “The more data is made openly available in a useful manner, the greater the level of transparency and reproducibility and hence the more efficient the scientific process becomes, to the benefit of society” (Molloy). The software certainly sounds like it will support this goal. As one writer puts it:

In terms of Open Science, my guess is that we’ll start to see authors publishing their articles straight onto a homepage or library repository, allowing for their work to be peer reviewed [through] almost instantaneously. Meanwhile, journals will likely operate in a post‐peer review niche, whereby they collect the most valuable articles and publish them in a context where they increase said article’s reputation value. (Winters)

Rosie Redfield and #arseniclife

We have already seen how an open peer‐reviewing model might work. One of the best‐known recent examples is in the work of the University of British Columbia’s Rosie Redfield. In 2010, when NASA‐funded scientists reported that they had found arsenic‐based life in California, Redfield, a microbiologist, took to her blog to express concerns about the data and the quality of the findings. Thanks to social media, her assessment spread rapidly among her scientific colleagues and in the press, casting doubt on NASA’s results (Zimmer).

What became known as the #arseniclife affair “is one of the first cases in which the scientific community openly vetted a high‐profile paper, and influenced how the public at large thought about it” (Zimmer). In 2011, Nature magazine named Redfield among the top 10 “people who mattered” for that year (Hayden).


Tools that allow internet users to more accurately assess information quality are a positive addition to our online lives. In the realm of scientific research and reporting, and similar products will inspire scientists to be forthcoming about their data and research methods, encourage journalists to take more care when describing and disseminating scientific findings, and give information professionals and consumers in general more confidence that what we are reading has been through some sort of transparent review and assessment process. I look forward to seeing this in practice.

[PS: Sign-up now for your username!]


Works Cited
Cox, R. “The Information Age and History: Looking Backward to See Us.” Ubiquity Sept. 2000. Web. 9 Sept. 2012.

Giles, J. “Truth Goggles.” New Scientist 15 Sept. 2012: 44‐47. Print.

Goldacre, B. “The Media Are Equally Guilty.” Guardian 20 Jan. 2010: 4. Web. 24 Sept. 2012.

‐‐‐. “When the Facts Get in the Way of a Story.” Significance 4.2 (7 June 2007): 84‐85. Web. 24 Sept. 2012.

Hayden, E. C. “365 Days: Nature’s 10. Ten People Who Mattered this Year. Rosie Redfield, Critical Enquirer.” Nature 480 (22 Dec. 2011): 437–445. Web. 29 Sept. 2012.

Molloy, J.C. “The Open Knowledge Foundation: Open Data Means Better Science.” PLOS Biology Dec. 2011. Web. 24 Sept. 2012.

Winters, J. “ The Future of the Internet and Peer Review.” 11 July 2012. Web. 1 Oct. 2012.

Zimmer, C. “The Discovery of Arsenic‐Based Twitter: How #arseniclife Changed Science.” Slate 27 May 2011. Web. 29 Sept. 2012.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s