🧑🏽🏫 Our insiders guide to social media research
How to get a quick idea of a field, evaluate a paper’s quality, and understand its findings
General learnings about academic research:
📑 Get a quick idea of a field of study from reading abstracts
🤼♂️ Evaluate a paper’s quality by assessing the peer review process of the journal it comes from, along with its methods and limitations
1️⃣ Begin to understand a paper’s findings by summarizing them in one sentence
Tech companies are always trying to optimize for human behavior. They have budgets to recruit employees with advanced degrees in social sciences like psychology, sociology, and communications studies. But what about those of us who don’t work at platforms and just want to understand social media better? It’s not easy to sift through the thousands of relevant scientific papers on Google Scholar, or even keep up with the new studies that regularly cascade around Twitter. Few studies make it into news articles, and when they do, much of the studies’ content is often lost.
Luckily, we have the perfect person to teach us how to read research like a pro: New_ Public Co-director Talia Stroud.
Talia is the founder and current director of the Center for Media Engagement and a professor at the University of Texas at Austin. “Broadly, I'm most interested in improving the media's democratic role,” says Talia. In addition to her PhD and award-winning book about how partisanship affects our media choices, Talia actively researches the effects of social media, including in our Signals framework for better digital public spaces.
Talia also teaches social media research and is a peer reviewer. In short, she reads a lot of this stuff, normally something like ten academic papers a week. Over Zoom, she walked me through several strategies to get more familiar with academic research on social media.
General tips for reading academic research
1. Read abstracts
Talia recommends starting by reading abstracts (an introductory, summary paragraph, typically on the first page, and often bolded) of lots of different research papers. “If you're just casually browsing and you want to kind of keep up, I think looking at abstracts gets you at least a taste of what's happening in the space,” says Talia. “And I think that would be a perfectly fine strategy if you just want a little bit of info to find out what's new.” After sampling a lot of abstracts, you can then be selective about which papers to read more closely.
2. Check out the source
Once you’ve decided to read a paper, here are some ways to assess its quality. It can’t be said enough: “It is important to pay attention to whether the article comes from a peer reviewed publication,” says Talia. “You should be critically aware of the sources that you're reading, because there are predatory journals out there.” While good research does sometimes end up in journals that don’t have as strong of a reputation, “the source is a heuristic.”
Talia says it’s usually easy to check the homepage of the journal for information about its review process. Beyond that, try Googling the name of the publisher and the phrase “peer review.” You can also kindly and respectfully ask researchers in the field — many of whom are on Twitter and are very friendly and helpful — if they’ve ever heard of the journal you’re looking at, and whether they have any insight into its quality.
3. Check out the methods
While trying to understand all the methodological details of academic research can be intimidating, there is often enough description in plain English to get a good picture of the study’s design. Talia says to remember that “there's no such thing as a perfect study, especially in the social sciences. This isn't like physics, where it's the same thing that tends to replicate under every single circumstance and it's always the same.”
Every study has weaknesses and limitations, and it’s useful to try and get a sense of those. Is the method reasonable for what the authors are trying to measure? Do they acknowledge the limits of what kinds of conclusions can be drawn from their work? Scientists are only truly confident in a new idea after performing many studies using different approaches and reaching similar conclusions.
4. Summarize the findings
Research papers often contain a ton of nuance and detail in their final sections. Talia says that she often will attempt to summarize the findings of a paper for herself. “When I'm reading ‘results’ sections, I'm trying to come away with the one or two sentences that I will actually remember about this article,” she says.
Putting the tips into action
To get a better sense of how to apply this advice, Talia took me through a paper published last fall entitled “The (minimal) persuasive advantage of political video over text.” This study, which seeks to measure how believable and persuasive political videos are compared to writing, is what Talia calls “a classic question,” and also extremely relevant to New_ Public. Let’s dig in!
Check out the source: I had to Google to find out that this journal, PNAS, is the Proceedings of the National Academy of Sciences. They have a lot of straightforward information about their peer review process on their website. Talia agrees that this is a highly reputable journal.
Check out the methods: In my reading, the experiment appears to be set up logically: researchers showed some people political ads or Covid-related videos and gave others a transcript of the videos to read instead, and then asked questions to measure how believable and persuasive the content was.
Talia agrees, saying the sample sizes are good, and that the analysis strategy makes sense. She notes that these aren’t random samples — these are people who signed up with a service to answer questions and earn points they can exchange for rewards. Talia says this practice is typical for a study like this and has its own pros and cons.
Talia also notes that while the content is the same between video and text, the text is based on video, which privileges the video. However, she acknowledges that it’s hard to imagine a more practical way to account for this in an experiment. “To be a true critic, you should propose an alternative, and recommend a way that it could have been done,” Talia says.
Also, the authors themselves note that field research is needed, outside of this experimental context. “The relative persuasive advantage of video versus text may in fact be even smaller outside an experimental context,” they write.
Summarize the findings: Talia summarizes this paper as, “video is more believable than text, but only slightly more persuasive.”
Expert insights
Beyond her general tips for me, Talia took me through the way she analyzed the paper and which parts were interesting to her. Here are some additional notes on “The (minimal) persuasive advantage of political video over text.”
Talia noticed that the authors categorized past research in this area as “inconclusive.” While this area isn’t her primary focus — so she can’t fully evaluate the claim — she says she found it interesting: “That's their entry point, right? That's why we need this new piece of research, because they're characterizing past work as being inconclusive.”
The paper talks about believability and persuasion separately. Talia wonders, “how distinct are these two?” and what their relationship is.
“It seems as though much of their work is on information, but they're also drawing conclusions about misinformation,” says Talia. “Does everything we know about the way in which people process and relate to information apply equally well to misinformation? Or do we process misinformation and think about misinformation in a different way? I could see a reasonable case on both sides of that.”
Thank you Talia! Check out her work at the Center for Media Engagement here. You can follow Talia and the Center on Twitter. We’d like to do a regular feature highlighting research here, so please send papers to us so can we discuss.
Joining our school communities design sprint
We’re doing a design sprint focused on a specific — and often frustrating — instance of gathering online: connecting adult caregivers with younger children in the same classroom or school, especially across class and race. In late April, our team will be gathering a group of parents and caregivers of children in Oakland, California, as well as design experts, technologists, and researchers, to consider how we might create digital spaces that support cross-connection and cohesion for caretakers of public school kids.
If this sounds like fun to you and you have passion, experience, or expertise to add, please let us know here. We can offer a small honorarium for your time.
Seeing and believing,
Josh
Illustration and design by Josh Kramer
New_ Public is a partnership between the Center for Media Engagement at the University of Texas, Austin, and the National Conference on Citizenship, and was incubated by New America.
 |
|