Content moderation

Content moderation is the process of monitoring, filtering and removing online user-generated content according to the rules of a private organization or the regulations of a government. It is used to restrict illegal or obscene content, spam, and content considered offensive or incongruous with the values of the moderator. When applied to dominant platforms with significant influence, content moderation may be conflated with censorship. Ethical issues involving content moderation include the psychological effects on content moderators, human and algorithmic bias in moderation, the trade-off between free speech and free association, and the impact of content moderation on minority groups.

Overview

Most types of moderation involve a top-down approach, where a moderator or small group of moderators are give discretionary power by a platform to approve or disapprove user-generated content. These moderators may be paid contractors or unpaid volunteers. A moderation hierarchy may exist or each moderator may have independent and absolute authority to make decisions.

In general, content moderation can be broken down into 6 major categories [1].

• Pre-Moderation screens each submission before it is visible to the public. This creates a bottleneck in user-engagement, and the delay may cause frustration in the user-base. However, it ensures maximum protection against undesired content, eliminating the risk of exposure to unsuspecting users. It is only practical for small user communities, and was common in moderated newsgroups on Usenet [2]

• Post-Moderation screens each submission after it is visible to the public. While preventing the bottleneck problem, it is still impractical for large user communities. Furthermore, as the content is often reviewed in a queue, undesired content may remain visible for an extended period of time, drowned out by benign content ahead of it, which must still be reviewed.

• Reactive moderation reviews only that content which has been flagged by users. It retains the benefits of both pre- and post-moderation, allowing for real-time user-engagement and the immediate review of only potentially undesired content. However, it is reliant on user participation and is still susceptible to benign content being falsely flagged. Most modern social media platforms, including Facebook and YouTube, rely on this method.

• Distributed moderation is an exception to the top-down approach. It instead gives the power of moderation to the users, often making use of a voting system. This is common on Reddit and Slashdot, the latter also using a meta-moderation system, in which users also rate the decisions of other users [3]. This method scales well across user-communities of all sizes, but also relies on users having the same perception of undesired content as the platform. It is also susceptible to groupthink and malicious coordination, also known as brigading [4].

• Automated moderation is the use of software to automatically assess content for desirability. It can be used in conjunction with any of the above moderation types. Its accuracy is dependent on the quality of its implementation, and it is susceptible to algorithmic bias and adversarial examples [5]. Copyright detection software on YouTube and spam filtering are examples of automated moderation [6].

• No moderation is the lack of moderation entirely. Such platforms are often hosts to illegal and obscene content, and typically operate outside the law, such as The Pirate Bay and Dark Web markets. Spam is a perennial problem for unmoderated platforms, but may be mitigated by other methods, such as limited posting frequency and monetary barriers to entry. However, small communities with shared values and few bad actors can also thrive under no moderation, like unmoderated Usenet newsgroups.

History

Pre-1993: Usenet and the Open Internet

Usenet emerged in the early 1980s as a network of university and private computers, and quickly became the world's first Internet community. The decentralized platform hosted a collection of message boards known as newsgroups. These newsgroups were small communities by modern standards, and consisted of like-minded, technologically-inclined users sharing the hacker ethic [7]. This collection of principles, including "access to computer should be unlimited", "mistrust authority; promote decentralization" and "information wants to be free", enabled a community that was mostly free of moderation. Because the network was distributed, it was resistant to top-down censorship, and only a minority of the newsgroups were moderated [8]. Users instead self-moderated and slow growth allowed new users to gradually become acclimated to its cultural norms, known as "netiquette" [9].

1994 - 2005: Eternal September and Growth

In September 1993, AOL offered access to Usenet to the general public, ushering in the Eternal September, a massive influx of users that did not share the same values as the founder population [10]. In 1994, the first instance of spam was recorded, instigating a massive backlash by the users [11]. In response, the first anti-spam bot was created, and the era of content moderation began [12].

With the invention of the World Wide Web, users began to drift away from Usenet and forums and blogs proliferated as replacements. These small communities had much stronger moderation than early Usenet groups, in response to the growth of spam and bad actors in the prevailing years. The distributed nature of these groups was such that, if a user did not like the moderation policies in one forum, they could easily move to another.

As platforms matured, they began to adopt limited content policies, in an ad-hoc manner. In 2000, Compuserve was the first platform to develop an "Acceptable Use" policy, which banned racist speech [13]. eBay soon followed in 2001, banning the sale of hate memorabilia and propaganda [14].

2006 - 2010: Social Media and Early Corporate Moderation

In the mid-2000s, social media platforms such as YouTube, Twitter, Tumblr, Reddit, and Facebook began to emerge, and quickly became massive solitary platforms that far outstripped in influence the old distributed blogs and message boards. These platforms initially struggled with content moderation. YouTube in particular developed ad-hoc rules from individual cases, gradually building up an internal set of rules that was opaque, arbitrary, and difficult for moderators to apply [15][16].

Other platforms, such as Twitter and Reddit initially adopted the unmoderated, free speech ethos of old, with Twitter claiming to be the "free speech wing of the free speech party" and Reddit stating that "distasteful" subreddits would not be removed, "even if we find it odious or if we personally condemn it" [17][18].

2010 - Present: Platform Dominance and Moderation Expansion

Throughout the 2010s, as the influence of social media platforms continued to grow and become more ubiquitous, the ethics of their moderation policies were brought into question. In having significant influence over national and international conversation, concerns about the presence of offensive content as well as concerns over the stifling of expression were raised [19]. Additionally, internet infrastructure providers also began to remove content hosted on their platforms.

In 2010, Wikileaks' leaked US Diplomatic Cables were removed by Amazon from their Cloud hosting service and their DNS provider also made the decision to drop their website [20].

In 2012, Reddit user /u/violentacrez was doxxed by Gawker, and the subsequent media spotlight causes Reddit to break with their previous stance, and ban /r/Creepshots for their controversial content. This led to further bannings for subreddits over the next few years, including /r/DeepFakes, /r/FatPeopleHate, and /r/GunsForSale.

In 2015, Instagram came under fire for moderating female nipples, which it viewed as obscene content, but not male nipples [21].

In 2016, in the aftermath of Gamergate and it's associated harrassment, Twitter instituted the Trust and Safety Council, also breaking with their previous free speech ethos [22].

In 2018, Tumblr banned adult content from their platform, leading to a mass removal of LGBT support groups and communities [23].