Algorithmic Bias and Prejudice

Algorithmic bias describe outcomes from computer systems “which are systematically less favorable to individuals within a particular group and where there is no relevant difference between groups that justifies such harms” . Algorithms, “which are a set of step-by-step instructions that computers follow to perform a task,” are commonplace in the 21st century and can cause various biases . There are multiple causes of bias in algorithms that can be classified into the following four groups: data creation bias, data analysis bias, data evaluation bias, and the demographics of data scientists. As a result of these biases, algorithms can create or perpetuate racism, sexism, and classism. To prevent those biases from a technical standpoint, it is important to monitor and analyze algorithms during data creation, analysis, and evaluation. From a general standpoint, it is important to increase diversity and inclusion in companies to mitigate biases from every aspect.

Causes of Bias

Data Creation Bias

Sampling Bias

Sampling bias is a type of bias that occurs when the dataset is created by choosing certain types of instances over others . This renders the dataset under representative of the population. An example of sampling bias is when a facial recognition algorithm is fed more photos of light-skinned people than dark-skinned people . This leads to the algorithm having a harder time recognizing darker-skinned people.

Measurement Bias

Measurement bias is a type of bias that occurs as a result of human error. It can occur when “proxies are used instead of true values” when creating datasets . An example of this type of bias was in the recidivism risk prediction tool. The tool used prior arrests and friend or family arrests as proxy variables to measure the level of riskiness or crime . Minority communities are policed more frequently, so they have higher arrest rates. The tool would conclude that minority communities have higher risk because they have higher arrest rates, but it does not take into account that there is a difference in how those communities are assessed .

Label Bias

Label bias is a type of bias that occurs as a result of inconsistencies in the labeling process . An example of this type of bias is two scientists labeling the same object as a stick and a twin rather than using the same label. Another type of label bias can occur when scientists assign a label based on their subjective beliefs rather than objective assessment .

Data Analysis Bias

Sample Selection Bias

Sample selection bias is a type of bias that occurs when the selection of data for analysis is not representative of the population being analyzed. An example would be when analyzing the effect of motherhood on wages. If the study is restricted to women who are already employed, the measured effect will be biased as a result of “conditioning on employed women” .

Confounding Bias

Confounding bias is a type of bias that occurs when the algorithm learns the wrong relations by not considering all the information or misses relevant relations . An example would be when graduate schools state that admissions are solely based on grade point average. There could other factors such as ability to afford tutoring which could depend on factors such as race. These factors could influence grade point average and thus admission rates. As a result, “spurious relations between inputs and outputs are introduced” which could lead to bias .

Data Evaluation Bias

Human Evaluation Bias

Human evaluation bias is a type of bias that occurs as a result of humans validating the performance of algorithms. Human beliefs can create bias during the evaluation of models .

Sample Treatment Bias

Sample treatment bias is a type of bias that occurs when the “test sets selected for evaluating an algorithm may be biased” . An example would be when showing an advertisement to a specific group of viewers, such as those speaking a certain language . The observed results would not be representative of the population as a whole since the algorithm was only tested on a specific group of people.

Impact

Racism

Algorithms have been criticized for perpetuating racism. One example is when a software could not identify Joy Buolamwini’s face. Buolamwini is a “Ghanaian- American graduate student at MIT” who “was working on a class project using facial-analysis software” . The software could not identify her face but could identify her light-skinned coworkers. Buolamwini “put on a white mask,” and the software was able to detect the “mask’s facial features perfectly” . Buolamwini learned that the software had been tested on a dataset that contained “78 percent male faces and 84 percent white faces” . Because the algorithm was trained on biased data, it was “forty-four times more likely to be misclassified than lighter-skinned males” .

Another example of racism in algorithms can be found when examining the health industry. Most hospitals use predictive algorithms to determine how to split up resources. An algorithm that was “designed to predict the cost of care as a proxy for health needs” consistently gave the “same risk score” to Black and white patients when the Black patient was far sicker than the white patient . This was “because providers spend much less on their care overall” . The algorithm was trained on historic data, and that history included “segregated hospital facilities, racist medical curricula, and unequal insurance structures, among other factors” . As a result, the algorithm consistently rated Black patients as costing less because history has valued Black patients less .

Sexism

One example of sexism in algorithms was discovered by Princeton University researchers. Using a machine learning software, they analyzed words . They found that the words “women” and “girl” were more likely to be “associated with the arts instead of science and math, which were most likely connected to males” . The algorithm had picked up on and incorporated human gender biases into its software. “If the learned associations of these algorithms were used as part of a search-engine ranking algorithm,” it could perpetuate and reinforce existing gender biases .

In 2012, Target used a “pregnancy detection model” to send coupons to women who were possibly pregnant in order to increase their profits . A father was outraged that his daughter was receiving pregnancy coupons when she was only a teenager, but he later learned that his daughter was in fact pregnant. As a result of Target’s algorithm, the teenager had lost “control over information related to her own body and her health” .

In 2018, Amazon was creating an algorithm to screen applicants. The dataset the algorithm was trained on included resumes from primarily men. Because of this, the algorithm “developed an even stronger preference for male applicants” and “downgraded resumes with the word women and graduates of women’s colleges” . The lack of representation in the dataset lead to a biased algorithm.