Content Moderation in Facebook

Workers at a Facebook content moderation center

Content moderation is the process of screening content users post online by applying a set of pre-set rules or guidelines to see if it is appropriate or not . Facebook is a social media platform that allows people to connect with friends, family and communities of people who share common interests . As with many other popular social media platforms, Facebook has come up with an approach to moderate and control the type of content users see and engage with. Facebook has two main approaches when it comes to content moderation; they utilize both AI moderators and human moderators. The way Facebook moderates its content is through its community standards that lay out what they believe each post should follow . Although most of the time Facebook’s content moderation tactics are successful in reducing the harmful content on its platform, people mainly focus on the instances where it has failed . The ethics behind Facebook’s content moderation approach has also been widely controversial from the mental health struggles human moderators are forced to deal with to questioning how the AI is trained to flag inappropriate content .

Overview/Background

How content is filtered

When it comes to content moderation, Facebook utilizes both AI moderators as well as human moderators. Posts that violate the community standards are deemed inappropriate. This includes everything from spam to hate speech to content that involves violence . Content that violates the community standards is found right away through the AI moderators, however, the human moderators are responsible for posts that the AI is not quite sure about and are vital to improving the machine learning model the AI technology uses .

AI Moderators

Facebook filters all posts initially through its AI technology. Facebook’s AI technology starts with building machine learning (ML) models that have the ability to analyze the content of a post or recognize different items in a photo. These models are used to determine whether what the post contains fits within the community standards or if there is a need to take action on the post, such as removing it .

Sometimes, the AI technology is unsure if content violates the community standards so it will send over the content to the human review teams. Once the review teams make a decision, the AI technology is able to learn and improve from that decision. This is how technology is trained over time and gets better. Oftentimes, Facebook’s community standard policies change to keep up with changes in social norms, language and our products and services. This requires the content review process to always be evolving and changing .

Until fairly recently, posts were viewed in the order they were reported, however, Facebook says it wants to make sure the most important posts are seen first. They decided to change their machine learning algorithms to prioritize more severe or harmful posts . Facebook has recently reworked how their deem posts a violation. They used to have separate classification systems that looked at individual parts of a post. They split it up into content type and violation type and would have many classifiers look at photos and text. Facebook decided it was too disconnected and created a new approach. Now, Facebook says their machine learning algorithm works through a holistic approach or Whole Post Integrity Embeddings (WPIE). It was trained on a very widespread selection of violations and has greatly improved

Human Moderators

Facebook filters all posts through its AI technology initially, but it is passed through to their human moderators if Facebook’s AI technology decides that certain pieces of content require further review. It will send the content to human review teams to take a closer look and make a decision on whether or not to remove the post. So, in other words, the human moderators get the final say .

There are about 15,000 Facebook content moderators employed throughout the world. Their main job is to sort through the AI flagged posts and make decisions about whether or not they violate the company’s guidelines . There are many companies that Facebook has worked with to help moderate content. Some of these companies included Cognizant, Accenture, Arvato, and Genpact .

Facebook is one of the largest social media platforms as the latest update from the Q3 2022 data reported 2.96 billion users worldwide . Since Facebook must cater to so many worldwide users, it is often when the human moderators are overwhelmed. Many human moderators have also expressed their struggles with mental health issues as a result of working that job. Many employees have reported the work enviornment being stressful, dirty and unhealthy .

Human moderators receive extremely low pay in comparison to Facebook's other employees. Human moderators at Cognizant, a company Facebook utilizes to provide moderators, earn just about $4 above the state's minimum wage. For example, employees of Cognizant who work in Pheonix, Arizona make only $28,800 per year. Whereas, the average Facebook employee has a total compensation of $240,000 . Many content moderators have expressed frustration with the low pay .

Self Moderation

Facebook additionally provides page managers tools with which to moderate their own pages, at a basic level these include what types of posts users can upload (photos, text, comments, nothing). These permissions extend to banning users, hiding or deleting comments and posts, and blocking profanity or specific vocabulary from text posts. These posts may not violate the community standards Facebook has created, but for whatever reason, the user does not want to see. Users are able to block certain words from being seen or block entire users entirely. While this option could be dangerous and create a biased or narrow-minded view of something, allowing the opportunity for users to moderate the content they see is important to Facebook. Facebook also has a "Moderation Assist" feature that uses AI to censor individual accounts or posts based on features of the account or post, including not having a profile picture, not having friends or followers, posts including links, or posts with custom keywords .

How Successful is Facebook’s Content Moderation Process?

There are over three million posts reported daily by either users or the AI screening technology as possible content guideline violations. In 2018, Mark Zuckerberg, the CEO of Facebook, stated that moderators “make the wrong call in more than one out of every 10 cases.” That equates to about 300,000 mistakes made everyday .

AI Mistakes

Facebook has faced criticism when it comes to their content moderation process. Specifically, the automated removal systems and the vague rules and unclear explanations of its decisions. Some reasons why the content moderation AI faces backlash is due to unreliable algorithms, vague standards/guidelines, not being able to see more context, and proportionality .

However, it is important to note that Facebook is constantly changing its guidelines to stay with the times. It is hard for the algorithm to change and update at the same pace .

Instances

Covid-19 (The Good & The Bad)

When the Covid-19 pandemic hit in March 2020, Facebook decided to send it's human moderators home and only rely on AI. As a result, Facebook saw a decrease in removal of posts when it came to child sexual abuse material and suicide and self-injury content. The drop in removal was not due to a decrease in those types of posts, but instead due to the limited number of actual humans who were available to look at those posts. Facebook also stated that they did not feel comfortable providing people with extremely graphic content at home .

Humans are not responsible to find all all child sexual abuse material. Automated systems are responsible for removing 97.5% of those types of posts that appear on Facebook. But if the human moderators aren't able to flag those types of posts, the AI system is unable to find it at scale. The Covid-19 Pandemic showed Facebook how vital and needed human moderators are in making sure their content moderation is a success .

Even though Facebook had access to a lesser amount of human moderators due to Covid-19, Facebook noted that it was able to improve in other areas through its AI technology. Some examples include improving its proactive detection rate for the moderation of hate speech, terrorism, and bullying and harassment content.

Ethical Concerns

Mental Health & Unfair Treatment of Human Moderators

One of the biggest ethical concerns when it comes to Facebook’s content moderation process is the human moderators. The human moderators are reported to have unfair wages, unfair hours, and are left to deal with many mental health issues .

Former moderators who worked for Facebook have come out to say how they have struggled with PTSD and trauma from some of the content they were forced to engage with . One former moderator, Josh Sklar, has stated how Facebook was always changing up the policies and guidelines and never really gave specific quota. In other words, he saw things as extremely mismanaged. He also expressed great frustration when it came to the type of content he was reviewing (such as child porn). He cited that as his main reason for quitting and hopes in the future, Facebook will have more emphasis on mental health for these moderators. .

Facebook uses content moderation provider companies from around the world and one of its largest has recently decided to discontinue working with Facebook. A TIME investigation found low pay, trauma and alleged union-busting surrounding the company .