In our previous article, we saw how the success of deep learning (or connectionist Artificial Intelligence (AI)) has renewed and complicated the question of the explainability of AI systems. By creating models that are often described as "black boxes", i.e. whose inner workings largely escape our comprehension, these techniques have made the need to understand and justify their results all the more crucial and arduous. This need, moreover, is driven ever more insistently by growing regulatory requirements such as the General Data Protection Regulation (GDPR) or the European AI Act (AI Act), and by a social demand for transparency and control over these systems.

Faced with this challenge, a specific field of research has emerged and experienced spectacular growth over the past ten years or so: Explainable AI (xAI). Its stated aim is to develop methods and techniques for making predictions, classifications or decisions derived from deep learning models intelligible. Today's xAI landscape is teeming with thousands of publications every year, proposing a multitude of approaches.

Despite this effervescence, one thing is clear: none of the explainability techniques produced seems to have reached a consensus, and the xAI field as a whole is struggling to achieve a stable scientific framework. Critics, both inside and outside xAI, point to a lack of formalism, problems of robustness, and difficulty in establishing consensual evaluation criteria. Why is xAI, despite its obvious importance, still a highly unstable scientific field?

To answer this question, this article proposes to go beyond a purely technical analysis of xAI methods. Drawing on the results of our sociological survey of the xAI field (Berkouk et. al 2024), we argue that the nature and limits of xAI are deeply linked to its institutional and functional position in relation to the field of study of connectionist AI and AI in general. Through the collection and analysis of over 16,000 scientific publications on SemanticScholar, we show that xAI seems to function as a "frontier space", one of whose effects would be to respond to external demands and criticisms addressed to AI in general. This field of research would thus support AI and facilitate its acceptance.

xAI: a field born to respond to external demands

Although the first reflections on the intelligibility of the results of AI models have existed at least since the early 90’s, a turning point is notable around 2015-2016, i.e. precisely during the period when connectionist techniques supplanted the symbolic paradigm.

The DARPA program: an engineering problem

Our analysis of the corpus of publications shows that the use of terms related to "explainability" ("Explainable AI", "xAI", etc.) undergoes a sudden growth from 2016 onwards, gradually supplanting the previously prevailing vocabulary of "interpretability". This change is not insignificant, and largely coincides with the launch of the Explainable AI program by DARPA (the US Defense Advanced Research Projects Agency) in 2015 (Gunning et al. 2021).

Figure 1 - Chronological evolution of the number of publications in the xAI field (Source: authors) 

With this program, DARPA's objective was to address a technical problem: faced with the rise of deep learning systems whose performance was impressive but whose operations were opaque, it was becoming crucial for critical applications (particularly military) to be able to understand, trust and manage these new tools. DARPA therefore sought to stimulate the development of techniques to overcome the opacity of neural network models, while maintaining high performance. xAI was thus born, in part, out of an institutional injunction to solve an engineering and operational confidence problem.

A rapid diversification of expectations and stakeholders

Very quickly, the motivations behind the development of xAI went well beyond this initial framework. An analysis of the motivations expressed by authors in the most cited publications in the field reveals a growing recognition of less technical expectations from an increasingly wide audience:

• user trust and acceptance: xAI is presented in the literature as a sine qua non condition for users to accept and adopt AI systems);
• regulatory and social requirements: the RGPD, with its articles on automated decision-making and the right to obtain useful information about their underlying logic, is as early as 2017, explicitly cited as a major motivation for developing techniques to make the use of "black boxes" compatible with its principles and obligations. The recent European AI Act only reinforces this trend;
• expectations in terms of social responsibility: concerns about the fairness, accountability, transparency and ethics of AI systems, relayed by the media and civil society, have become important drivers of xAI research.

Figure 2 - Diagram showing the different aims and addressees of explicability techniques, from Barredo Arrieta et al. 2020.

Thus, xAI appears less as a field defined by a clearly delimited internal scientific question, than as a space tasked with responding to a growing and heterogeneous set of external demands aimed at making connectionist AI socially acceptable and legally compliant.

Heterogeneous, non-consensual solutions

This heavy dependence on external demands has direct consequences on the way xAI structures itself and on the nature of its scientific output.

A proliferation of heterogeneous solutions

Faced with the diversity of expectations (to explain in order to debug, to trust, to verify compliance, to challenge a decision...), the xAI field has produced a multitude of explanation techniques that are very different from one another (see our second article on xAI techniques). As we saw in our first article on the renewal of the issue of explainability, explaining a connectionist system is an operation necessarily ex post to its training.

However, this proliferation is often to the detriment of scientific rigor and consolidation. As pointed out by critics from within the field, xAI suffers from a lack of clear definitions, unified formalisms and standardized evaluation metrics. Many techniques have been shown to be fragile or even manipulable, and theoretical results even show that some popular methods may, in some cases, be no more likely than a random explanation to understand the actual influence of a model parameter (Bilodeau et al. 2024).

Yet, unlike in other fields (such as the study of algorithmic fairness, or fairness, in AI where impossibility theorems have structured and shaped the ways in which questions are addressed in this field, cf. Friedler et al . 2021) these fundamental limitations seem to have a limited impact on the practice of xAI research. The field seems to prioritize the production of new methods in response to new demands, rather than the critical deepening and consolidation of existing knowledge.

A striking symptom of this situation is the exceptionally high proportion of literature reviews (surveys) in xAI publications. The aim of a literature review is to collect, analyze and organize scientific articles and content, in order to provide an overall view of scientific advances in a given field. Our comparative analysis shows that the "Explainable AI" topic contains proportionally at least five times as many reviews as other comparable topics in computer science. These literature reviews attempt to bring order to the heterogeneity of the field by proposing taxonomies to classify the different methods. However, far from establishing a consensus, these taxonomies are themselves in competition with each other: classification criteria vary from one review to another, a sign that the field is struggling to agree on its own foundations. And rather than providing a lasting structure for the field, literature reviews seem to have the primary function of "mapping" the solutions on offer for external players (researchers from other fields, industry, regulators) seeking to find their bearings in this complex landscape.

Figure 3 - Example of a taxonomy from Linardatos et al. 2020

xAI as a "subordinate frontier space

How can we explain this particular dynamic? Our hypothesis is that the characteristics of the xAI derive from its subordinate position within the connectionist AI ecosystem. The xAI would function as a frontier space, at once distinct from the core of AI (model development) but essential to its expansion.

A division of scientific labor

Analysis of the trajectories of the most influential researchers in the xAI field reveals a form of division of labor, i.e. a distribution of tasks and research questions between certain groups of researchers:

The major players in xAI often come from fields other than AI: A significant proportion (nearly 60% in our analysis of the 100 most cited authors) of the most renowned xAI researchers had no publications in the field of deep learning before becoming involved in xAI. They come from other areas of computer science, or even from application fields such as medicine (the second most productive discipline in xAI after computer science).

AI "stars" have little presence in xAI: Conversely, the most cited and recognized researchers in the field of neural network development and deep learning are only marginally involved in xAI research. Of the 100 most-cited authors in the field of AI, only 10 appear in our xAI database, and only one is among the authors of the 100 most-cited xAI publications.

So, while the "core" of AI research is focused on improving model performance, the xAI research community seems to remain exogenous, composite, striving to meet the expectations and injunctions of a plurality of stakeholders.

The xAI "shield" function

It is this marginal position, yet essential to the acceptability of AI, that effectively places the xAI field in the role of a subordinate space. Its main function, observed on a collective scale (as sociological analysis allows), is not so much the development of an "autonomous" science of explanation, as that of an interface with the outside world. It should be stressed here that this is not a conscious or intentional strategy on the part of xAI researchers themselves: it is indeed the "social" relationship between the field of AI and the field of xAI that gives rise in this field to production dynamics aimed at absorbing criticism, responding to the demands of regulators and providing the tools - however imperfect - to reassure and justify deployments. Researchers insert themselves into these pre-existing and collective production logics, enabling the core of AI to continue developing, sheltered from the most direct controversies. At the same time, this "shield" position limits their opportunities to participate in the formalization of a scientific framework for the field.

Figure 4 - xAI's "challenges" to achieve "responsible AI", from Barredo Arrieta et al. 2020.

If this socio-political function were confirmed, it could explain xAI's focus on overflowing production of responses to external demands, and its difficulty in reaching consensus on problems, methods and evaluation criteria, and thus defining a stable scientific framework.

Our sociological analysis of the xAI field shows that explainability cannot be reduced to a mere technical challenge. To understand it, we need to look beyond the technical artefacts and take a close look at the players behind it, the institutions that support and fund it, and the practical work involved in devising and developing technical solutions to meet specific and sometimes contradictory expectations. To ignore this social dimension would be to miss an important part of what constitutes the explainability of AI today, and to potentially accept solutions that only superficially address the legitimate concerns it raises.

For public authorities, whose mission is to help define a framework for artificial intelligence, adopting such a perspective is of major strategic interest. It allows us to understand that future standards of explainability - what we consider to be a "good" explanation - are not born ex nihilo at the moment of formal regulation. They are built upstream, at the very heart of the scientific research and development process. And this process is influenced by the expectations of a wide range of players, particularly those in the business world. Understanding these contextual mechanisms of co-production of explainability standards is crucial to politically questioning the objectives assigned to these techniques, submitting them to democratic debate and ensuring that the regulatory framework does not simply endorse standards considered to be neutral.

Summary ⮕

⬅ Previous article [2/3]