š§š½āš« Our insiders guide to social media research
How to get a quick idea of a field, evaluate a paperās quality, and understand its findings
General learnings about academic research:
š Get a quick idea of a field of study from reading abstractsĀ
š¤¼āāļø Evaluate a paperās quality by assessing the peer review process of the journal it comes from, along with its methods and limitations
1ļøā£ Begin to understand a paperās findings by summarizing them in one sentence
Tech companies are always trying to optimize for human behavior. They have budgets to recruit employees with advanced degrees in social sciences like psychology, sociology, and communications studies. But what about those of us who donāt work at platforms and just want to understand social media better? Itās not easy to sift through the thousands of relevant scientific papers on Google Scholar, or even keep up with the new studies that regularly cascade around Twitter. Few studies make it into news articles, and when they do, much of the studiesā content is often lost.
Luckily, we have the perfect person to teach us how to read research like a pro: New_ Public Co-director Talia Stroud.Ā
Talia is the founder and current director of the Center for Media Engagement and a professor at the University of Texas at Austin. āBroadly, I'm most interested in improving the media's democratic role,ā says Talia. In addition to her PhD and award-winning book about how partisanship affects our media choices, Talia actively researches the effects of social media, including in our Signals framework for better digital public spaces.


Talia also teaches social media research and is a peer reviewer. In short, she reads a lot of this stuff, normally something like ten academic papers a week. Over Zoom, she walked me through several strategies to get more familiar with academic research on social media.
General tips for reading academic research
1. Read abstracts
Talia recommends starting by reading abstracts (an introductory, summary paragraph, typically on the first page, and often bolded) of lots of different research papers. āIf you're just casually browsing and you want to kind of keep up, I think looking at abstracts gets you at least a taste of what's happening in the space,ā says Talia. āAnd I think that would be a perfectly fine strategy if you just want a little bit of info to find out what's new.ā After sampling a lot of abstracts, you can then be selective about which papers to read more closely.
2. Check out the source
Once youāve decided to read a paper, here are some ways to assess its quality. It canāt be said enough: āIt is important to pay attention to whether the article comes from a peer reviewed publication,ā says Talia. āYou should be critically aware of the sources that you're reading, because there are predatory journals out there.ā While good research does sometimes end up in journals that donāt have as strong of a reputation, āthe source is a heuristic.ā
Talia says itās usually easy to check the homepage of the journal for information about its review process. Beyond that, try Googling the name of the publisher and the phrase āpeer review.ā You can also kindly and respectfully ask researchers in the field ā many of whom are on Twitter and are very friendly and helpful ā if theyāve ever heard of the journal youāre looking at, and whether they have any insight into its quality.
3. Check out the methods
While trying to understand all the methodological details of academic research can be intimidating, there is often enough description in plain English to get a good picture of the studyās design. Talia says to remember that āāāthere's no such thing as a perfect study, especially in the social sciences. This isn't like physics, where it's the same thing that tends to replicate under every single circumstance and it's always the same.āĀ
Every study has weaknesses and limitations, and itās useful to try and get a sense of those. Is the method reasonable for what the authors are trying to measure? Do they acknowledge the limits of what kinds of conclusions can be drawn from their work? Scientists are only truly confident in a new idea after performing many studies using different approaches and reaching similar conclusions.
4. Summarize the findings
Research papers often contain a ton of nuance and detail in their final sections. Talia says that she often will attempt to summarize the findings of a paper for herself. āWhen I'm reading āresultsā sections, I'm trying to come away with the one or two sentences that I will actually remember about this article,ā she says.
Putting the tips into action
To get a better sense of how to apply this advice, Talia took me through a paper published last fall entitled āThe (minimal) persuasive advantage of political video over text.ā This study, which seeks to measure how believable and persuasive political videos are compared to writing, is what Talia calls āa classic question,ā and also extremely relevant to New_ Public. Letās dig in!
Check out the source: I had to Google to find out that this journal, PNAS, is the Proceedings of the National Academy of Sciences. They have a lot of straightforward information about their peer review process on their website. Talia agrees that this is a highly reputable journal.
Check out the methods: In my reading, the experiment appears to be set up logically: researchers showed some people political ads or Covid-related videos and gave others a transcript of the videos to read instead, and then asked questions to measure how believable and persuasive the content was.Ā
Talia agrees, saying the sample sizes are good, and that the analysis strategy makes sense. She notes that these arenāt random samples ā these are people who signed up with a service to answer questions and earn points they can exchange for rewards. Talia says this practice is typical for a study like this and has its own pros and cons.Ā
Talia also notes that while the content is the same between video and text, the text is based on video, which privileges the video. However, she acknowledges that itās hard to imagine a more practical way to account for this in an experiment. āTo be a true critic, you should propose an alternative, and recommend a way that it could have been done,ā Talia says.
Also, the authors themselves note that field research is needed, outside of this experimental context. āThe relative persuasive advantage of video versus text may in fact be even smaller outside an experimental context,ā they write.
Summarize the findings: Talia summarizes this paper as, āvideo is more believable than text, but only slightly more persuasive.ā
Expert insightsĀ
Beyond her general tips for me, Talia took me through the way she analyzed the paper and which parts were interesting to her. Here are some additional notes on āThe (minimal) persuasive advantage of political video over text.ā
Talia noticed that the authors categorized past research in this area as āinconclusive.ā While this area isnāt her primary focusĀ ā so she canāt fully evaluate the claim ā she says she found it interesting: āThat's their entry point, right? That's why we need this new piece of research, because they're characterizing past work as being inconclusive.ā
The paper talks about believability and persuasion separately. Talia wonders, āhow distinct are these two?ā and what their relationship is.Ā
āIt seems as though much of their work is on information, but they're also drawing conclusions about misinformation,ā says Talia. āDoes everything we know about the way in which people process and relate to information apply equally well to misinformation? Or do we process misinformation and think about misinformation in a different way? I could see a reasonable case on both sides of that.ā
Thank you Talia! Check out her work at the Center for Media Engagement here. You can follow Talia and the Center on Twitter. Weād like to do a regular feature highlighting research here, so please send papers to us so can we discuss.
Joining our school communities design sprint
Weāre doing a design sprint focused on a specific ā and often frustrating ā instance of gathering online: connecting adult caregivers with younger children in the same classroom or school, especially across class and race. In late April, our team will be gathering a group of parents and caregivers of children in Oakland, California, as well as design experts, technologists, and researchers, to consider how we might create digital spaces that support cross-connection and cohesion for caretakers of public school kids.Ā
If this sounds like fun to you and you have passion, experience, or expertise to add, please let us know here. We can offer a small honorarium for your time.
Seeing and believing,
Josh
Illustration and design by Josh Kramer
New_ Public is a partnership between the Center for Media Engagement at the University of Texas, Austin, and the National Conference on Citizenship, and was incubated by New America.