Content moderation

Content moderation is the process of monitoring, filtering and removing online user-generated content according to the rules of a private organization or the regulations of a government. It is used to restrict illegal or obscene content, spam, and content considered offensive or incongruous with the values of the moderator. When applied to dominant platforms with significant influence, content moderation may be conflated with censorship. Ethical issues involving content moderation include the psychological effects on content moderators, human and algorithmic bias in moderation, the trade-off between free speech and free association, and the impact of content moderation on minority groups.

Overview

Most types of moderation involve a top-down approach, where a moderator or small group of moderators are give discretionary power by a platform to approve or disapprove user-generated content. These moderators may be paid contractors or unpaid volunteers. A moderation hierarchy may exist or each moderator may have independent and absolute authority to make decisions.

In general, content moderation can be broken down into 6 major categories.

• Pre-Moderation screens each submission before it is visible to the public. This creates a bottleneck in user-engagement, and the delay may cause frustration in the user-base. However, it ensures maximum protection against undesired content, eliminating the risk of exposure to unsuspecting users. It is only practical for small user communities, and was common in moderated newsgroups on Usenet.

• Post-Moderation screens each submission after it is visible to the public. While preventing the bottleneck problem, it is still impractical for large user communities. Furthermore, as the content is often reviewed in a queue, undesired content may remain visible for an extended period of time, drowned out by benign content ahead of it, which must still be reviewed.

• Reactive moderation reviews only that content which has been flagged by users. It retains the benefits of both pre- and post-moderation, allowing for real-time user-engagement and the immediate review of only potentially undesired content. However, it is reliant on user participation and is still susceptible to benign content being falsely flagged. Most modern social media platforms, including Facebook and YouTube, rely on this method.

• Distributed moderation is an exception to the top-down approach. It instead gives the power of moderation to the users, often making use of a voting system. This is common on Reddit and Slashdot, the latter also using a meta-moderation system, in which users also rate the decisions of other users. This method scales well across user-communities of all sizes, but also relies on users having the same perception of undesired content as the platform. It is also susceptible to groupthink and malicious coordination, also known as brigading.

• Automated moderation is the use of software to automatically assess content for desirability. It can be used in conjunction with any of the above moderation types. Its accuracy is dependent on the quality of its implementation, and it is susceptible to algorithmic bias and adversarial examples. Copyright detection software on YouTube and spam filtering are examples of automated moderation.

• No moderation is the lack of moderation entirely. Such platforms are often hosts to illegal and obscene content, and typically operate outside the law, such as The Pirate Bay and Dark Web markets. Spam is a perennial problem for unmoderated platforms, but may be mitigated by other methods, such as limited posting frequency and monetary barriers to entry. However, small communities with shared values and few bad actors can also thrive under no moderation, like unmoderated Usenet newsgroups.

History

Pre-1993: Usenet and the Open Internet

Usenet emerged in the early 1980s as a network of university and private computers, and quickly became the world's first Internet community. The decentralized platform hosted a collection of message boards known as newsgroups. These newsgroups were small communities by modern standards, and consisted of like-minded, technologically-inclined users sharing the hacker ethic. This collection of principles, including "access to computers should be unlimited", "mistrust authority: promote decentralization", and "information wants to be free", created a culture that was resistant to moderation and free of top-down censorship. The default assumptions were of users acting in good faith and that new users could be gradually assimilated into the shared culture. As a result, only a minority of newsgroups were moderated, most allowing anyone to post however they pleased, as long as they followed the community's social norms, known as "netiquette." Furthermore, the Internet in general was considered separate and distinct from the physical space its servers were located, existing in its own "cyberspace" not subject to the will of the state. Throughout this era of the Open Internet, online activity mostly escaped the notice of government regulation, creating a policy gap that only began to close in the late 1990's.

1994 - 2005: Eternal September and Growth

In September 1993, AOL began offering Internet access to the general public. The resulting flood of users arrived too quickly and in too many numbers to be assimilated into the existing culture, and the shared values that had allowed unmoderated newsgroups to flourish were lost. This was known as the Eternal September, and the resulting growth transformed the Internet from a high-trust to a low-trust community. The consequences of this transformation were first seen in 1994, when the first recorded instance of spam was sent out across Usenet. The spam outraged Usenet users, and the first anti-spam bot was created in response, ushering in the era of content moderation.

With the invention of the World Wide Web, users began to drift away from Usenet, while thousands of forums and blogs emerged as replacements. These small communities were often overseen by single individuals or small teams, and exercised total moderating control over their domains. In response to the growth of spam and other bad actors, these often had much stricter rules than early Usenet groups. However, the vast marketplace of available forums and places of discussion was such that, if a user did not like the moderation policies in one platform, they could easily move to another.

As corporate platforms matured, they began to adopt limited content policies as well, though in a more ad-hoc manner. In 2000, Compuserve was the first platform to develop an "Acceptable Use" policy, which banned racist speech eBay soon followed in 2001, banning the sale of hate memorabilia and propaganda.

2006 - 2010: Social Media and Early Corporate Moderation

In the mid-2000s, social media platforms such as YouTube, Twitter, Tumblr, Reddit, and Facebook began to emerge, and quickly became dominant, centralized platforms that gradually displaced the multitude of blogs and message boards as a place for user discussion. These platforms initially struggled with content moderation. YouTube in particular developed ad-hoc policies from individual cases, gradually building up an internal set of rules that was opaque, arbitrary, and difficult for moderators to apply.

Other platforms, such as Twitter and Reddit adopted the unmoderated, free speech ethos of old, with Twitter claiming to be the "free speech wing of the free speech party" and Reddit stating that "distasteful" subreddits would not be removed, "even if we find it odious or if we personally condemn it."

2010 - Present: Centralization and Expanded Moderation

Throughout the 2010s, as social media platforms became ubiquitous, the ethics of their moderation policies were brought into question. As these centralized platforms began to have significant influence over national and international discourse, concerns were raised overthe presence of offensive content as well stifling of expression. Additionally, internet infrastructure providers also began to remove content hosted on their platforms.

In 2010, Wikileaks leaked the US Diplomatic Cables and hosted the documents on Amazon Web Services. These were later removed by Amazon as against their content policies. WikiLeak's DNS provider also made the decision to drop their website, effectively removing WikiLeaks from the Internet until an alternative host could be found.

In 2012, Reddit user /u/violentacrez was doxxed by Gawker Media for moderating several controversial subreddits, including /r/Creepshots. The subsequent media spotlight caused Reddit to reconsider their minimalist approach to content moderation. This set a precedent which was used to ban more subreddits over the next few years. In 2015, Reddit banned /r/FatPeopleHate, which marked a turning point at which Reddit no longer considered itself a "bastion of free speech." In 2019, Reddit banned /r/WatchPeopleDie, in an effort to suppress the spread of the Christchurch mass shooting, a move widely considered as censorship.

In 2015, Instagram came under fire for moderating female nipples, which it viewed as obscene content, but not male nipples.

In 2016, in the aftermath of Gamergate and it's associated harrassment, Twitter instituted the Trust and Safety Council, also breaking with their previous free speech ethos.

In 2018, Tumblr banned adult content from their platform, leading to a mass removal of LGBT support groups and communities.