Algorithms have been rising fast and saturating our modern world, from simple ones generating early credit scores in the 1960s to deep neural networks identifying security risks such as guns or certain blacklisted individuals in CCTV feeds. The rise of algorithms is motivated by both operational cost savings and higher quality decisions. Human decision-makers often are more expensive than a machine, especially when thousands or millions of similar decisions need to be made, and we’re notoriously fickle, inconsistent deciders.
To achieve this efficiency and consistency, algorithms are designed to remove many human cognitive biases, such as confirmation bias, overconfidence, anchoring on irrelevant reference points (which mislead our conscious reasoning), and social and interest biases (which cause us to override proper reasoning due to competing personal interests).
Yet in designing algorithms, we could go even further. The algorithms we implement could become tools to help tackle even deeper-seated societal biases, such as notorious racial and gender biases.
Historically, algorithms haven’t quite lived up to this promise. In fact, very often the opposite has been true.
As an avalanche of news and research reports has shown, algorithms still are not free of bias and often exacerbate it. While sometimes the effects of algorithmic bias are trivial (such as our social media feeds being anchored in puppy videos because the very first post you ever clicked on was one), at other times they can wreak havoc on a person’s life.
The algorithms we implement could become tools to help tackle even deeper-seated societal biases, such as notorious racial and gender biases.
Often, specific (not least demographic) groups of individuals are affected. COMPAS, an algorithm U.S. authorities use to estimate how likely it is that a criminal will re-offend, has been found to exhibit racial bias; Google’s algorithms for picking job ads have shown a preference for lower-paying jobs for female users. In cases like these, algorithms aren’t merely failing to correct bias; they’re entrenching it.
While some algorithmic biases are artifacts of human error—created by inadequate data or statistical techniques, for instance—many algorithmic biases mirror societal biases at large. A number of studies have made the case that bail and parole judges show bias if the defendant is Black or the judge is tired; biased policing (e.g., in deciding to pull over a vehicle for a routine check or whether to carry out a drug test) can create biased evidence. Actions taken by these biased actors will feed into algorithms; if a re-offender is more likely to actually get convicted of another offence if she is Black, the algorithm will assign a higher probability of re-offending to Black people.
Worryingly, sometimes we can’t even recognize that an algorithm is biased until real-life circumstances change drastically. For example, orchestras typically did not even consider female applicants, but then it became common practice to hide auditioning musicians behind curtains, thus concealing their gender. Now female musicians abound in the world’s leading orchestras—but if their pay is decided by humans, gender discrimination still can persist.
For governments and business managers, then, expunging algorithmic bias becomes a circular problem. The only data they have to develop algorithms is shaped by the very bias they are fighting. What’s more, data scientists have insufficient control over the hundreds or thousands of agents that generate the data and therefore imbue it with their own deeply rooted biases.
For governments and business managers, then, expunging algorithmic bias becomes a circular problem. The only data they have to develop algorithms is shaped by the very bias they are fighting.
How can we build algorithms that correct for biased data and that live up to the promise of equitable decision-making?
When we consider changing an algorithm to eliminate bias, it is helpful to distinguish what we can change at three different levels (from least to most technical): the decision algorithm, formula inputs, and the formula itself.
In discussing the levels, I will use a fictional example, involving Martians and Zeta Reticulans. I do this because picking a real-life example would, in fact, be stereotyping—I would perpetuate the very biases I try to fight by reiterating a simplified version of the world, and every time I state that a particular group of people is disadvantaged, I also can negatively affect the self-perception of people who consider themselves members of these groups. I do apologize if I unintentionally insult any Martians reading this article!
On the simplest and least technical level, we would adjust only the overall decision algorithm that takes one or more statistical formulas (typically to predict unknown outcomes such as academic success, recidivation, or marital bliss) as an input and applies rules to translate the predictions of these formulas into decisions (e.g., by comparing predictions with externally chosen cutoff values or contextually picking one prediction over another). Such rules can be adjusted without touching the statistical formulas themselves.
How can we build algorithms that correct for biased data and that live up to the promise of equitable decision-making?
An example of such an intervention is called boxing. Imagine you have a score of astrological ability. The astrological ability score is a key criterion for shortlisting candidates for the Interplanetary Economic Forecasting Institute. You would have no objective reason to believe that Martians are any less apt at prognosticating white noise than Zeta Reticulans; however, due to racial prejudice in our galaxy, Martian children tend to get asked a lot less for their opinion and therefore have a lot less practice in gabbing than Zeta Reticulans, and as a result only one percent of Martian applicants achieve the minimum score required to be hired for the Interplanetary Economic Forecasting Institute as compared to three percent of Zeta Reticulans.
Boxing would posit that for hiring decisions to be neutral of race, for each race two percent of applicants should be eligible, and boxing would achieve it by calibrating different cut-off scores (i.e., different implied probabilities of astrological success) for Martians and Zeta Reticulans.
Another example of a level-one adjustment would be to use multiple rank-ordering scores and to admit everyone who achieves a high score on any one of them. This approach is particularly well suited if you have different methods of assessment at your disposal, but each method implies a particular bias against one or more subsegments. An example for a crude version of this approach is admissions to medical school in Germany, where routes include college grades, a qualitative assessment through an interview, and a waitlist.
At the second level, moving up in technical complexity, you could adjust inputs into the statistical formula. In credit scoring, there exists a precedent insofar as many countries require personal bankruptcies “to be forgotten” after a certain number of years. Expanding on this (and staying with our hiring example), you can analytically determine which inputs into your astrological ability score show systematic differences for historically discriminated populations, and mathematically adjust them. For example, say speed shouting while staring without blinking is what you care about. Martians tend to achieve a 20 percent lower test score in speed shouting than Zeta Reticulans but can stare at you without blinking 10 percent longer than their galactical counterparts. If both metrics enter your formula, in the most basic form you could divide the speed shouting test score by 0.8 and the stare score by 1.1 for Martian applicants in order for them to look on average like Zeta Reticulans (and it goes without saying that more sophisticated adjustment approaches exist).
On the third, and most technical, level, you could develop a statistical formula with an explicit switch to turn off a source of bias (such as race). This approach is most elegant but also most anathema to standard statistical procedures (that, by definition, are grounded in reality as opposed to wishful thinking). Imagine you included race in an interplanetary score of astrological ability. The algorithm would estimate the probability that a Martian correctly predict the future and the probability that a Zeta Reticulan did. With a statistical sleight of hand, it can adjust for these differences and set everyone’s race to Martian, thus eliminating any effect of race. (Technical note: in order to completely remove the bias, you need to remove the effect of race from all other inputs—this then means for the algorithm that the race variable also accounts for expected differences in speed shouting and staring ability.)
This practice of simulating different states of the world is, in fact, widely used in some disciplines such as economics. It is also very transparent, and if you think of the estimated probability as a score (similar to, say, point scores used for immigration admissions in some countries), setting a Martian’s race flag to that of a privileged race is the same as giving some bonus points to a disadvantaged group.
Algorithms that eliminate systematic biases create environments that encourage disadvantaged groups to succeed.
Why do these techniques matter? They confer at least three advantages. First, they are based in an explicit calibration to reach a neutral ground—a bias manifested in the data against a particular group is eliminated but there is no advantage given to anyone beyond that. This heads off complaints about reverse discrimination.
Second, there is ample evidence that credible judgments can create their own reality. Just as the famous experiment by Jane Elliott showed that school children will perform better if they are told that they are genetically destined to be more intelligent, a score vouching for an applicant’s ability could become a self-fulfilling prophecy. Algorithms that eliminate systematic biases create environments that encourage disadvantaged groups to succeed.
And third, they are easy to administer. One big challenge to overcoming human bias in decision-making is that you need to overcome bias one decision-maker at a time (e.g., by providing anti-bias training to recruiters or managers responsible for staff evaluations). The reason algorithms can be a game-changer in fighting bias is that they are centrally designed and deployed through automated systems—thus forcefully overcoming widespread cognitive biases. Doesn’t this sound like a powerful formula for a better world?
