[2/3] Believing or doubting: the issue of confidence bias in decision-making

This article is the second in a series of three :

• Human supervision, hybrid decisions: what are the challenges?  
• Believing or doubting: the issue of confidence bias in decision-making  
• Predicting without explaining, or when algorithmic opacity muddies the waters

In December 2023, the Court of Justice of the European Union ruled that the credit scoring tool of the German company SCHUFA, which provided banks with an estimate of a client’s creditworthiness in the form of a trust score, constituted a fully automated decision. In practice, however, the decision to grant credit was formally made by a bank employee responsible for verifying and either applying or rejecting the tool’s proposal, thereby implying human intervention. Yet, since the score was never actually contested by bank employees, who systematically relied on it, effectively turning the suggestion into the credit decision, the Court concluded that such intervention was not meaningful.

To be more than a mere formality, human oversight, as defined in the European AI Act, must make it possible to detect errors, diverge from, or interrupt the system ; in other words, to provide a genuine alternative to the AI system’s output. These abilities, however, rely on psychological dispositions highlighted by the regulation, which specifies that the person responsible for oversight must be aware of potential cognitive biases, such as excessive trust. This bias poses a risk of undermining the quality of the oversight exercised, and consequently, the reliability of the entire mechanism.

Empirical literature does not point to a uniform reaction of individuals toward algorithms but instead identifies two tendencies : either an acceptance heuristic (the appreciation thesis), noted in the European AI Act, or a rejection heuristic (the aversion thesis), despite the errors that both strategies can produce.

Too much trust brings risk: excessive appreciation

Some studies highlight a tendency among participants to conform to the system’s outputs (e.g. Jacobs et al. 2021, Green 2019, Yin 2019, Bussone et al. 2015, Kiani et al. 2020, Alberdi et al. 2004, Logg et al. 2019, Robinette et al. 2017). In such cases, human oversight ceases to be effective, as the decision-maker systematically relies on the system’s output, mirroring the logic of the Court of Justice of the European Union’s ruling on SCHUFA. Here, it is not the effectiveness of the decision itself that is in question, but rather the effectiveness of human oversight over algorithmic decision-making, and its ability to reject potential system errors or anomalies.

In a study conducted in psychiatry (Jacobs et al. 2021), researchers asked a cohort of clinicians to make a series of decisions about a fictitious patient. For each patient, the clinician had to decide on a treatment, either with a recommendation from a machine learning system accompanied by a brief justification of the suggested choice, or entirely without any recommendation (a completely independent decision).

The study shows that, on the one hand, the performance of the groups with and without access to the machine learning system’s recommendations is virtually the same, and that both groups make poorer decisions than the system alone. On the other hand, when the system produces an incorrect decision (where the error is defined as a position diverging from that of a panel of psychiatry experts), performance drops compared to the control group (without access to the machine learning system) that is, humans tend to conform to the algorithm’s decision in such cases.

Finally, the authors observed an effect of familiarity with the tool : clinicians who were more familiar with the system were, on average, less likely to follow a machine learning system’s recommendation, regardless of its accuracy, compared to clinicians who were less familiar with such systems.

The observation that the level of professional expertise plays a role in decision-making is corroborated by a study by Gaube et al. 2021 in which two groups of doctors with different levels of medical expertise were asked to produce a diagnosis and rate a recommendation displayed as coming from either an algorithm or a human, when in fact all the recommendations came from a human.

Doctors in the most expert group tend to rate recommendations less highly when they are indicated as coming from an AI system. However, the quality of their diagnosis is influenced by the quality of the advice received, regardless of its source (AI or human).

This result suggests that the influential effects of the system's advice could be a simple artifact of not having been able to form a decision before the confrontation, and not necessarily due to an attitude of deference toward the algorithm. Indeed, the study shows that participants generally tended to follow the advice given, whether it came from a human or a machine.

Absence of deference to the system by experts has also been observed in other studies (Logg et al. 2020, Povyakalo et al. 2013). In the latter study, assessment was also modulated by disagreement. Advice that contradicted the participant's prior opinion had less impact but did not completely cancel out the effect of trust in the system. The assessment of algorithms decreased (but did not disappear) when their advice was contrary to their own judgment. The authors conclude that it is on these points of disagreement that people are most likely to improve their accuracy. These critical cases of confrontation are therefore interesting for decision-making.

Aversion and excessive mistrust

Some studies suggest excessive mistrust of the system, preventing participants from making informed decisions (e.g. Dietvorst et al. 2015, Longoni et al. 2019, Dzindolet et al. 2002, Lim et O’Connor 1995, Yeomans et al. 2019, Promberger et Baron 2006).

In several forecasting tasks where they had to choose between an algorithmic prediction and a human prediction regarding the likelihood of a song's success (Dietvorst et al. 2015), when they saw the system working, and sometimes getting it wrong, participants tended to reject its predictions in favor of human advice, despite the higher error rate of human predictions (up to twice that of the algorithm).

In short, humans are less forgiving of algorithms' mistakes and tend to generalize their overall performance based on these harmful examples. This tendency persists even after observing that the system's performance exceeds that of humans on average. A natural explanation is that mistrust of algorithms exists even before observing them, and that observing mistakes reinforces this preconception.

There would therefore appear to be an anchoring bias at work: once users have formed an opinion about an AI system, it is very difficult for them to change their minds, even in the face of contradictory evidence. This is supported by the fact that aversion to algorithms is not observed in purely deterministic tasks such as logical calculations or memory tasks, where humans are notoriously more fallible than algorithms.

Amplification of biases and harmful interactions

Humans are not free from bias and do not have infallible judgment, which leads to errors that, all other things being equal, are added to those of the systems or amplify them.