Science

Hey all, thanks for reporting this to bring some extra attention to it. I'm going to leave this article up, as it is not exactly misinformation or anything otherwise antithetical to being shared on this community, but I do want to note that there are four different sources here:

• There's the original study which designed the misinformation susceptibility test; the ArXiv link was already provided, but in case anyone would like a look the study was indeed peer reviewed and published (as open access) in the journal Behavior Research Methods. As with all science, when reading the paper it's important to recognize exactly what it is the authors were even trying to do, taking into account that they're likely using field-specific jargon. I'm not a researcher in the social sciences so I'm unqualified to have too strong an opinion, but from what I can tell they did achieve what they were trying to with this study. There are likely valid critiques to be made here, but as has already been pointed out in our comments many aspects of this test were thought out and deliberately chosen, e.g. the choice to use only headlines in the test (as opposed to, e.g., headlines along with sources or pictures). One important thing to note about this study is that it is currently only validated in the US. The researchers themselves have made it clear in the paper that results based on the current set of questions likely cannot be compared between countries.

• There's the survey hosted on streamlit. This is being run by several authors on the original paper, but it is unclear exactly what they're going to do with the data. The survey makes reference to the published paper so the data from this survey doesn't seem like it was used in constructing the original paper (and indeed the original paper discusses several different versions of the test as well as a longitudinal study of participants). Again, taken for what it is I think it's fine. In fact I think that the fact that this survey has been made available is why this has generated so much discussion and (warranted) skepticism. Being able to test yourself on a typical survey gives a feel for what is and isn't actually being measured. I consider this a pretty good piece of science communication / outreach, if nothing else.

• There is the poll by YouGov. This is separate from the original study. The researchers seem to be aware of it, but as far as I can tell weren't directly involved in running the poll, analyzing the data, or writing the article about it. This is not inherently a bad poll, but I do think it's worth noting that it is not a peer reviewed study. We have little visibility into how they conducted their data analysis here, for one thing. ~~From what I can tell without knowing how they actually did their analysis the data here looks fine, but (this not being a scientific paper) some of the text surrounding the data is a bit misleading.~~ EDIT: Actually it looks like they've shared their full dataset including how they broke categories down for analysis, it's available here. Seeing this doesn't much change my overall impression of the survey other than to agree with Panteleimon that the demographic representation here is not very well balanced, especially once you start trying to take the intersections of multiple categories. Doing that, some of their data points are going to have much lower statistical significance than other. My main concern is that some of the text surrounding the data is kinda misleading. For example, in one spot they write, "Older adults perform better than younger adults when it comes to the Misinformation Susceptibility Test," which (if their data and analysis can be believed) is true. However nearby they write, "Younger Americans are less skilled than older adults at identifying real from fake news," which is a different claim and as far as I can tell isn't well supported by their data. To see the difference, note that when identifying real vs fake news a reader has more to go on than just a headline. MIST doesn't test the ability to incorporate all of that context, that's just not what it was designed to do.

• Finally, there's the linked phys.org article. This is the part that seems most objectionable to me. The headline is misleading in the same way I just discussed, and the text of the article does a bad job of making it clear that the YouGov poll is different from the original study. The distinction is mentioned in one paragraph, but the rest of the article blends quotes from the researchers with YouGov polling results, strongly implying that the YouGov poll was run by these researchers (again, it wasn't). It's a bit unfortunate that this is what was linked here, since I think it's the least useful of these four sources, but it's also not surprising since this kind of pop-sci reporting will always be much more visible than the research it's based on. (And to be clear, I feel I could have easily linked this article myself, I probably wouldn't have even noticed the conflation of different sources if this hadn't generated so many comments and even a report; just a good reminder to keep our skeptic hats on when we're dealing with secondary sources.)

Finally, I'd just like to say I'm pretty impressed by the level of skepticism, critical thinking, and analysis you all have already done in the comments. I think that this indicates a pretty healthy relationship to science communication. (If anything folks are maybe erring a bit on the side of too skeptical, but I blame the phys-org article for that, since it mixed all the sources together.)