How comments affect whether we see bias in a news story

Michele Jones, then a PhD student at the University of North Carolina, presented preliminary findings at the Convergence and Society conference in South Carolina in October 2008 on a question news sites had not thought to ask: whether the comments under a story change how the story itself is read. The brief was two-part — what comments do to the perceived credibility of an article, and what they do to the perceived bias in it — and the second half is what the study is remembered for.

The design was deliberately narrow. An online experiment put a single news article about abortion in front of 76 participants. The article did not change between conditions; the comments attached to it did, carrying either pro-life or pro-choice views. That isolates one variable, which is what makes the result readable at all: any difference in how the article was judged had to come from the discussion around it rather than from the reporting.

The first result reproduced something already well established. Readers holding strong views perceived the article as biased against their own side — the hostile media effect, documented across numerous studies, in which partisans on opposite sides of the same issue both read the same coverage as tilted against them. The second result was the interesting one, because it cut against expectation: statistical analysis showed the comments did not, by themselves, make readers see more bias in the story. On the measure the experiment was built to test, the comment thread left the article where it found it.

What complicated that was what participants wrote when they were invited to add their own thoughts at the end of the study. Those open responses suggested the character of the comments did affect the reading, but indirectly, and in a way a bias scale would not pick up. A reader might judge the article balanced and still start wondering whether the journalist had picked the pro-life or the pro-choice comments deliberately. The suspicion attached to the selection rather than to the writing — that is, to an editorial decision the reader could not see, made about material the reader could.

Jones treated the study as a starting point rather than a conclusion, and said so: the useful questions it opened were about the relationship between a news story and the discussion around it. At the time, news outlets were experimenting with several forms of user-generated content at once and attaching them to articles without much evidence about the effect, which is why the finding was presented as an argument for further study rather than as guidance. The question has only become more pressing since, as comment, quoting and reposting have moved off the article page entirely and onto platforms the publisher does not run.