Black Box Algorithms

Introduction

A black box algorithm is created directly from data by an algorithm, meaning that humans, even those who design them, cannot understand how variables are combined to make predictions. Even if one has a list of the input variables, black box predictive models can be such complicated functions of the variables that no human can understand how the variables are jointly related to each other to reach a final prediction. These algorithms are in areas such as criminal justice, finance, and healthcare. Their use has raised many important ethical and legal questions. One of the main concerns about black box algorithms is their potential for bias and discrimination. Since the algorithm's inner workings are not transparent, it can be challenging to detect and correct any biases that the algorithm may have encoded into the system. These biases are perceived to lead to unfair and unjust outcomes, particularly for marginalized groups such as people of color and low-income individuals. According to researchers at duke, "what stands between the status quo and that idealistic future is not more data or more code, but less bias in data and code."

Examples of Black Box Algorithms with Problems

Google

Google's Black Box Algorithm refers to the company's proprietary search engine ranking system, and Google has not fully disclosed this system to the public. Google's algorithm is widely recognizable for its effective delivery of relevant and high-quality search results. The algorithm considers hundreds of factors to determine the relevance and importance of a website and its content. These factors include keyword usage, backlinks, website structure, and user engagement metrics. However, the exact algorithm and weight given to each factor are not publicly known, making it difficult for website owners and marketers to understand why a particular site ranks higher or lower in the search results. As a result, some experts have raised concerns about the potential for bias and discrimination in the algorithm, as it is not fully known by the public how the algorithm takes into account factors such as race, gender, and political views.

One of Google's facial recognition algorithms mistakenly identified two Black customers in a photo as gorillas in 2015, Google was the subject of a fiery uproar. Within hours of the error being discovered on Twitter, they announced a remedy, and the scandal eventually went away. A few years later, a journalist went back to the issue and discovered something unexpected: Google had no way of identifying gorillas. In other words, Google's programmers have just eliminated the system's ability to label items as gorillas at all, rather than coding in a remedy. Google's response was undoubtedly a band-aid to get out of a sticky PR situation, but it also highlights how erratic and unpredictable algorithms are. They frequently behave in unexpected ways and are challenging to examine.

COMPAS

As technology advances in our society, more and more decisions affecting individuals are being made by hidden algorithms. One area where this has raised concern is in the criminal justice system, specifically regarding the fairness and due process of these algorithms. One example of a widely-used, yet secretive algorithm is COMPAS (which stands for Correctional Offender Management Profiling for Alternative Sanctions) has been the focus of many legal challenges. COMPAS is a an algorithm that is frequently employed in the criminal justice system that is intended to calculate the risk that offenders would reoffend. In order to help ensure public safety and lower recidivism, the algorithm is used to help guide choices concerning bail, sentence, and parole.

COMPAS was made to try and lessen prejudice and discrimination in the criminal justice system. The algorithm can help in ensuring that choices concerning bail, sentencing, and parole are based on objective criteria rather than on subjective judgments or bias views by employing a standardized, data-driven approach. By doing this, you may make sure that criminals receive proper punishment and that the judicial system if fairly treating everyone despite their differences. It is thought that the effectiveness of the criminal justice system can be improved by COMPAS. The algorithm can help to ensure that offenders are placed in the proper degree of supervision and that resources are directed to those who are most likely to reoffend by providing a more precise and unbiased estimate of the risk of recidivism. By doing so, you can improve public safety and lower the overall cost of the criminal justice system. Despite these advantages, there are significant drawbacks to using COMPAS. The algorithm's lack of transparency and difficulty in comprehending how it generates predictions is one of the primary issues. Due to this, it may be challenging to verify if the algorithm is impartial and free from prejudice.

Loomis v. Wisconsin

When deciding if the defendant in Loomis v. Wisconsin should be sentenced to life in prison without the possibility of parole, the trial court took into account the state's employment of a closed-source risk assessment tool to assess the defendant's likelihood of reoffending. There are ethical concerns about fairness, openness, and the possibility of prejudice when closed-source risk assessment tools are used in criminal sentencing. Closed-source tools may be opaque, according to critics, making it challenging for defendants and their attorneys to comprehend the considerations that went into the assessment and contest the validity of the conclusions. Moreover, there are worries that risk assessment tools' closed-source nature may also imply that they are not exposed to the same amount of scrutiny and testing as open-source tools, which could produce unreliable or skewed results.

In preparation for sentencing, a Wisconsin Department of Corrections officer produced a PSI that included a COMPAS risk assessment. COMPAS assessments estimate the risk of recidivism based on both an interview with the offender and information from the offender’s criminal history. As the methodology behind COMPAS is a trade secret, only the estimates of recidivism risk are reported to the court. At Loomis’s sentencing hearing, the trial court referred to the COMPAS assessment in its sentencing determination and, based in part on this assessment, sentenced Loomis to six years of imprisonment and five years of extended supervision.

Medical Algorithms

New machine-learning techniques entering medical practice are often both opaque and changeable, raising challenges in validation, regulation, and integration into practice. These conventional validation tools fail for black-box algorithms because neither programmers nor users are aware of the precise measurements that an algorithm makes, or more precisely, the constellation of previously measured characteristics that it takes into account, or the biomedical quantities that it tracks, but only what it predicts or suggests. Although it only offers a little degree of assurance, scientific understanding does provide some, and black-box algorithms lack this assurance. These predictions and suggestions may also alter over time when the algorithm is modified to take into account fresh data, which complicates the validation process. Black-box algorithms frequently cannot rely on scientific understanding to offer baseline confidence in their efficacy because to this lack of explicit knowledge, and they do not themselves improve understanding.

By focusing on clinical trials, black-box medicine may also help with the development of new pharmaceuticals or the clinical verification of unorthodox uses for currently available medicines. Clinical trials can already be made more efficient by using pharmacogenomic testing and explicit tailored medicine. Through the recommendation of participant groups that satisfy a more complicated set of criteria for as-yet-unknown reasons, black-box medicine may further widen these options. Additionally, black-box medications could significantly lower the expense of clinical trials by avoiding more of them, particularly in the context of above-described off-label applications for medicines that have already received approval.

White Box Models vs Black Box Models

Black Box vs White Box

Black Box	White Box
It is a way of software testing in which the internal structure or the program or the code is hidden and nothing is known about it.	It is a way of testing the software in which the tester has knowledge about the internal structure or the code or the program of the software.
Implementation of code is not needed for black box testing.	Code implementation is necessary for white box testing.
No knowledge of implementation is needed.	Knowledge of implementation is required.
No knowledge of programming is required.	It is mandatory to have knowledge of programming.
It is least time consuming.	It is most time consuming.

[1]

The Harvard Data Science Review wrote an article about the Explainable Machine Learning Challenge. The goal of the competition was to create a complicated black box model for a certain dataset and explain how it functioned.Their article tries to figure out whether the real world of machine learning is similar to the Explainable Machine Learning Challenge, where black box models are used even when they are not needed. The 2018 Explainable Machine Learning Challenge serves as a case study for considering the tradeoffs of favoring black box models over interpretable ones.[2]

Advantages of Black Box Algorithms

←Back • ↑Topics • ↑Categories

Massive amounts of data can be used to train black box algorithms, and if that data is internalized, they can make decisions intuitively or experientially like people. This indicates that computers are now capable of finding dynamic solutions to issues based on patterns in data that humans may not even be able to notice, rather than simply following explicit pre-written instructions. [3]

Deep neural networks, for instance, are a type of machine-learning system that can be just as challenging to comprehend as the human brain. The decision-making mechanism of these intricate networks of artificial neurons cannot be easily mapped out. In higher dimensional spaces, where humans are unable to see them, other machine-learning algorithms are capable of detecting geometric patterns.[4]

Additionally, these algorithms are being used to automate consequential decisions, such as lending, assessing job applications, informing release on parole and prescribing life altering medications.[5]

Black box medicine has the ability to bring huge benefits to health care systems. Black-box personalized medicine is key to realizing the next generation health benefits of genomics, electronic health records, and big data in the health care sector.[6] Black-box personalized medicine can help resolve a major challenge facing the drug industry: When can already-approved drugs be prescribed or used for a new purpose? It is extremely costly to develop new drugs.

Ethical Concerns

Bias

In addition to inheriting many of the best traits of the human brain, artificial intelligence has also demonstrated that it knows how to use it skillfully and ably. Object recognition, map navigation, and speech translation are just a few of the many skills that modern AI programs have mastered, and the list will not stop growing anytime soon. [7] Unfortunately, as a result of this it has amplified the bias that humans have. AI can help identify and reduce the impact of human biases, but it can also make the problem worse by baking in and deploying biases at scale in sensitive application areas. [8] For example, Propublica analyzed data from Broward County Florida and they claimed that the prediction model used there (COMPAS) is racially biased.

The first person to bring up examples of racial prejudice in picture categorization algorithms was Dr. Stacy Tantum, Bell-Rhodes Associate Professor of the Practice of Electrical and Computer Engineering. Tantum contends that early facial recognition failed to accurately identify people with darker skin tones because the underlying training data, or observations used to guide the model's learning process, did not adequately represent all skin tones. She emphasized the value of model openness further, saying that an engineer cannot fairly claim that an AI is objective if they regard it as a "black box" or a decision-making process that does not require explanation.[9]

Accountability

Another concern is that black box algorithms may lack accountability. Since it is difficult to understand how the algorithm arrives at its predictions, it can be difficult to hold those who created or use the algorithm responsible for any negative consequences. This can make it difficult to ensure that the algorithm is being used ethically and in compliance with laws and regulations.[10]

Designers and users of algorithms are typically blamed when problems arise. Blame can only be justifiably attributed when the actor has some degree of control and intentionality in carrying out the action.Algorithms “are opaque in the sense that if one is a recipient of the output of the algorithm (the classification decision), rarely does one have any concrete sense of how or why a particular classification has been arrived at from inputs”[11]

The merging of algorithm explainability with privacy was described by Dr. David Hoffman, the Steed Family Professor of the Practice of Cybersecurity Policy at the Sanford School of Public Policy. He cited the development of regulatory laws in other nations that provide limitations, responsibility, and oversight of personal data in cybersecurity applications. Said Hoffman, “If we can’t answer the privacy question, we can’t put appropriate controls and protections in place.”[12]

Transparency

One of the main ethical concerns about black box algorithms is their lack of transparency.[13] They are called Black box algorithms because they are opaque, meaning it is basically impossible to understand how they came up with any of their predictions or decisions. This can make it challenging to ensure that the algorithm is fair and unbiased and that it is not unintentionally causing harm to specific groups of people. For example, in the medical field, if they cannot figure out where reliable knowledge is coming from, should they blindly believe that it is correct? There is a growing concern that the black-box nature of some algorithms makes it impossible to believe in the reliability of the algorithm, and because of this, researchers, physicians, and patients need to find out if they can trust the results of such systems.[14] Biological systems are so complex, and big-data techniques are so opaque, that it can be difficult or impossible to know if an algorithmic conclusion is incomplete, inaccurate, or biased. And these problems can arise due to data limitations, analytical limitations, or even intentional interference.[15]

There is still much research to be done to understand when and how best to act responsibly and be transparent about the algorithms being built. Deciding what to disclose is just a start; the communication vehicle also needs to be explored. Human-computer interaction as well as machine learning and software engineering have roles to play here as well.[16]