Content moderation avoidance strategies
Authors: Rachel E. Moran, Kolina Koltai, Izzi Grasso, Joseph Schafer, Connor Klentschy (University of Washington’s Center for an Informed Public)
Key Takeaways:
Vaccine-opposed communities circumvent the community guidelines and moderation features of social media platforms through a variety of means. These include lexical variation (such as using “V@cc1ne”), covering up potentially rule-triggering images and text, and using ephemeral platform features such as Instagram stories to spread vaccine misinformation.
This analysis highlights just a few of the avoidance tactics used on Facebook, Twitter, and Instagram. The full range and extent of these strategies is difficult to systematically track, in part, because they are constantly evolving.
Actions taken by platforms to remove COVID-19 vaccine misinformation fail to counter the range of avoidance strategies vaccine-opposed groups deploy. We recommend that moderation efforts to remove vaccine misinformation look beyond content-based strategies to consider the accounts and communities that create and deploy moderation avoidance strategies.
In recent months, social media sites like Facebook, Instagram, and Twitter have taken significant steps to reduce the amount of vaccine-related misinformation on their platforms through the banning of anti-vaccination groups, take-downs of accounts of prominent anti-vaccine advocates, and the moderation of vaccine opposed misinformation. While these efforts are vital, misinformation around the COVID-19 vaccine is still abundant, and anti-vaccine advocates are still able to use these social media platforms to spread vaccine hesitancy.
In order to avoid moderation, vaccine-opposed communities have developed a range of tactics that they believe reduces the chances of their content being flagged by automated content moderation practices. The most common of these is lexical variation—the practice of changing how a keyword or phrase is spelled while still conveying its original meaning without triggering keyword moderation. Commonly used variations include “V@ccine,” “Vak-Seen,” and “V@X.” Not only does the proliferation of these terms assist in evading moderation, it also makes it difficult for researchers and public health communicators to understand the full picture of vaccine-related misinformation being shared.
Moderation-avoidance strategies go far beyond just lexical variation. During the course of the Virality Project we have noted a range of visual and technical strategies both deliberately and unintentionally deployed by anti-vaccine influencers to spread misinformation without platform detection. The following analysis documents a broad range of lexical, technical, and visual techniques used to spread COVID-19 vaccine misinformation across Twitter, Instagram, and Facebook.
We emphasize that community-driven practices to avoid detection and/or moderation are not endemic to anti-vaccination conversation; many of these practices have been developed by Internet communities around the globe to afford them free and safe speech on digital platforms and can thus be used to spread vital and socially responsible information that may be deemed politically controversial. In light of this, the following analysis makes recommendations to platforms specifically about moderation of vaccine-related misinformation. Further, we highlight how these practices illuminate the uncertainty surrounding the extent of vaccine-related misinformation online and the blindspots researchers and public health communicators may have in assessing the full extent of the problem.
Techniques used to Circumvent Pl@tf0rm M0der@tion P0licies
Through a qualitative content analysis of social media content shared by “repeat offenders” (accounts that regularly post vaccine-opposed misinformation) we highlight a number of different technical, lexical, and visual practices that have been utilized to spread vaccine misinformation and counter moderation policies, on three different platforms; Twitter, Facebook, and Instagram (see Table 1). We discuss the popular tactics used on each platform separately, but these tactics were often documented across multiple platforms.
Table 1: Moderation-avoidance strategies used on major platforms to spread vaccine-hesitant content.
Visual blocking of vaccine-related keywords and images
Perhaps the most pervasive moderation-avoidance strategy after lexical variation of keywords is the visual blocking of keywords in images. Users deploy emojis, images or even simply black out words they believe may trigger automated moderation. Often this blocking is poorly done—it is not only easy to tell from context what is blocked but often the word is still somewhat visible, as shown in the leftmost image in Figure 1. The use of vaccine emojis and pathogen cartoons signals to anti-vaccine community members that they are part of what they deem to be a “censored” conversation.
Links to vaccination misinformation included in comment section rather than in-post
Anti-vaccine information sharers posit that user comments are less likely to be moderated than the post itself given that comments are usually moderated by Page owners rather than platforms. To take advantage of this perceived loophole, posters use the comments sections of a benign post to link to externally hosted misinformation, or set-up misleading context in the main post that they support with an external link shared through the comments (see center image in Figure 1).
Sharing screenshots of anti-vaccination articles rather than links
In a similar vein, anti-vaccine content is often spread through screenshots rather than with direct links, a tactic also thought to avoid automated moderation. Instead of posting a direct link to an outside website containing misinformation (or even legitimate news stories), users post a screenshot of the article or website they wish to share content from and post the screenshot on their pages. This allows users to add in misleading contextual information at the top of the post or as a caption (see image on right in Figure 1) or direct users off-platform to unreliable news sources.
Figure 1: A page reposts a tweet by conservative commentator Candace Owens using red lines to attempt to block out “COVID-19” and “vaccine” (left). A user links to a story on heart inflammation and COVID-vaccines in the comments of an anecdote shared about a young man’s “regret” following a vaccination (center). A screenshot of a New York Times article on a FDA announcement related to the Johnson & Johnson vaccine shared alongside comments that vaccinated individuals are “lab rats” (right). *Select an individual image to enlarge.
Ephemeral content strategies
On Instagram, ephemeral content containing falsehoods and/or misinformation can still do a lot of damage (even if it is short-lived) due to platform affordances that enable users to take screenshots of the story, repost a user’s story, or share the story with other users. Anti-vaccine information sharers have propagated “folk theories” of how to avoid getting shadowbanned or taken down from Instagram by sharing content through Instagram stories, as opposed to the Instagram feed. Users believe that automated content-moderation features respond primarily to in-post content, so posting more controversial content on the stories feature means it is less likely to be flagged by the platform. Users also claim that sharing “normal” content, e.g., outfit pictures, scenery, food, etc., between stories on more controversial topics also works to minimize platform moderation and the effectiveness of other users “flagging” the Story content to Instagram as problematic (see image on left in Figure 2).
Using coded language
Another tactic used by members of the anti-vaccine community, especially female-identifying persons, is alluding to COVID-19 vaccines using terms such as “toxins,” “pharmaceuticals,” “carcinogens,” etc. This has been particularly successful in spreading vaccine-hesitant narratives linking the vaccine to women’s health issues (see center image inFigure 2).
Linktrees and “Link in Bio”
Linktree is a hosting website popularly used by influencers to gather together their various different social media and website links in one place. Instagram users place a Linktree URL in their bio which redirects users to an external Linktree page where they can collate their links. Users of this feature often refer audiences of their posts and stories to the “link in their bio.” By taking users outside of Instagram’s “walled garden,” Linktree allows users to draw their audiences to content beyond the reach of Instagram’s community guidelines or moderation systems. Consequently, using Linktree (and linking to vaccine-hesitant content in Instagram bios) has become a popular content-moderation avoidance strategy for anti-vaccine communities (see image on right in Figure 2).
Figure 2: An Instagram user shares a Story and links to a video of actor and anti-vaccination advocate Rob Schneider arguing against mandatory vaccination and shares on the next slide “normal” content—a link to buy similar outdoor lounge chairs (left). An Instagram post suggesting that the COVID-19 vaccine has negative impacts for women’s health. Notably the post does not explicitly use “vaccine” or “COVID-19” but instead refers to “toxins” (center). A Linktree website linked from a known anti-vaccine advocate. Note the links to different conspiratorial content, vaccine-hesitant and anti-vaccine content. (right). *Select an individual image to enlarge.
Twitter
Lexical variation of keywords
Common across all platforms, and particularly present on Twitter, is the use of lexical variations of keywords related to the COVID-19 vaccine. Users vary the spelling of words like COVID, Vaccine, Vaccination, Jab—often using emojis or symbols to represent missing letters. For example, to refer to the vaccine, terms like “v@x,” “the jab”, or even
“v🪓ine” (using the Ax emoticon to represent the “axe” sound in “vaccine”) are substituted (see image to the left in Figure 3). These variations have proliferated, making it difficult to maintain a cohesive list of commonly used terms. Further, anti-vaccination influencers with consistent followings develop novel vocabularies that extend beyond lexical variation, for example referring to the vaccine as a “serum” or even simply “the thing,” with their audiences understanding the implied connection.
Screenshotting and re-sharing previously deleted tweets as images
Even when content is taken down by platforms, it can retain a second life through screenshots taken before the content was removed. On Twitter, this often means sharing a screenshotted tweet along with a comment that implies the content, or the individual who originally shared it, has been censored. Users believe that sharing a screenshotted image version of the content (see middle image in Figure 3) evades the algorithmic moderation of keywords that applies to written content.
Using vaccine-neutral hashtags to spread misinformation within pro-vaccination conversations
Finally, we see instances of Twitter users opposed to vaccines using pro-vaccine or vaccine-neutral hashtags in order to get their messages to a wider audience and to avoid the moderation they see associated with anti-vaccine hashtags (see image to the right in Figure 3). Posters using this tactic tend to present content with ambiguous framing, meaning that only those with knowledge of the poster, or who are well-versed in the talking points of the anti-vaccine community, would read the tweet as being anti-vaccine.
Figure 3: An example of lexical variation. This post is unlabeled, and, over a week after being posted, is still available on Twitter (left). A user shared a screenshot of a previously deleted tweet by Dr. Naomi Wolf who was banned from Twitter for sharing vaccine and COVID-19 misinformation (center). Prominent anti-vaccine influencer Dr. Sherri Tenpenny, uses a vaccine-neutral hashtag (#COVIDshot) in order to get her tweet which casts doubt on a vaccine program in Georgia to more users (right). *Select an individual image to enlarge.
Conclusion
The tactics explored above are by no means exhaustive. Indeed, we see similar visual and lexical tactics used on YouTube and TikTok as well as other social media platforms that require further analysis to properly document. What this initial exploration highlights, however, is the difficulty for researchers and public health communicators to accurately document the spread of vaccine-related misinformation on social media. Further, the prevalence of these tactics, and their seeming success, highlights failures in current attempts to curb the spread of vaccine-opposed content. A content-based approach to moderation fails to account for the ways in which vaccine-opposed communities build strategies to overcome moderation. Therefore it’s important to be aware of these blindspots and evolve moderation approaches as the anti-vaccine communities evolve their tactics.