Explainability, misrepresentation, and the commercialization of artificial intelligence

Explainability (and its sibling, interpretability) is arguably the most important new legal concept developed in response to the creation of highly complex artificial intelligence models trained on massive amounts of data and computer power. As is now well known, some of these models, such as the most sophisticated large language models, have inference mechanisms that are so mathematically complex that we cannot explain them. These “black box” models have been used to produce impressive natural language outputs and photorealistic images and video that are revolutionizing the “last mile” of technical solution delivery in numerous applications and industries. On the other hand, “black box” models may also face resistance from commercial users who are concerned with risk management. Regulators are also highly attuned to the risks of such models. This is where explainability enters the picture.

Explainability serves as a cornerstone of AI regulation

Explainability matters because users have a higher level of trust and greater perception of fairness when algorithmic outputs and decisions are explained in a way that the user can understand. Law makers and regulators argue that explainability is democratic and required for the rule of law. Explainability is democratic because it ensures transparency. Explainability enables the rule of law because it subjects algorithmic outputs to the principle of reasoned justification. Law makers and regulators argue that the combined effect enables effective oversight of artificial intelligence systems.

This is why explainability is a cornerstone of AI regulation, from the EU AI Act, to Canada’s Artificial Intelligence and Data Act, and to regulatory agency risk management guidelines and frameworks in Canada, the United States, Europe, the UK, and globally.

Explainability also serves a commercial function. Sophisticated explainability will enable the commercialization of high risk use cases because it allows for user-level active risk management as well as audit and higher level governance oversight by all stakeholders. It will also reduce the risk of end users making incorrect assumptions about outputs or misunderstanding the inferences that generated those outputs, leading to a chain of poor decisions and outcomes.

Explainability is also a central component of developing standards and benchmarks for AI systems to promote and proliferate trustworthy AI, and enable commercialization through universal ISO-type standards. The NIST framework in the United States is a good example of these efforts. The NIST Artificial Intelligence Risk Management Framework addresses explainability: “explainability refers to a representation of the mechanisms underlying AI systems’ operation, whereas interpretability refers to the meaning of AI systems’ output in the context of their designed functional purposes. Together explainability and interpretability assist those operating or overseeing an AI system, as well as users of an AI system, to gain deeper insights into the functionality and trustworthiness of the system, including its outputs.”

There are various approaches to develop governance and audit frameworks that comply with the explainability requirements in legislation and regulation depending on the regulatory and commercial context. These can include certain documentation and reporting mechanisms and governance frameworks. However, just like AI itself, explainability is an evolving concept.

What might the future of explainability look like?

Currently, there is commentary in the academic community that large language models express the same essential limitations as humans for certain cognitive tasks, and display similar biases such as priming, size congruity, and time delays. These biases and inaccuracies are often undetectable by humans simply relying on an output generated by an AI system, particularly very complex systems built from neural-net deep learning applied to massive data sets.

To solve these problems, researchers are currently exploring a number of solutions. Intelligent AI agents, for example, can be designed for the purpose of helping humans interact with and understand other artificial intelligence systems. They do so by, for example, reducing human user cognitive load, which can improve the quality of decisions and the ability of that user to understand an AI system’s outputs. Such intelligent agents can also be designed to explicitly provide explanations about why other AI systems are making certain recommendations or decisions, thereby improving human-AI collaboration and comprehension. Large language models can be used to improve the end user’s understanding of an underlying system by being engineered specifically to, for example, not just produce or draw inferences from data but also to explain in natural language how data outputs were derived. In the future these explanations will not necessarily need to take written form, but could also involve the creation of augmented reality environments designed to facilitate human understanding and improve human reasoning working with AI systems. These types of agents will be intended to improve human acceptance of AI outputs, reduce complaints and skepticism, and ultimately produce higher quality combined human-AI outputs.

Symbolic AI may also contribute to solve the explainability problem and enhance the safety of higher risk use cases for artificial intelligence. Symbolic AI involves building a fundamental knowledge base from which to build deep learning systems and otherwise validate those systems. This knowledge base generally constitutes a set of axioms to represent certain constants. These databases can serve as a source of truth to reject hallucinations, bias large language models towards correctness rather than inaccuracy, expand large language model coverage to solve for a wider set of problems using validated axioms and logical relationships, and provide audit provenance support for AI System risk management. Research in this area is also exploring the construction of neuro-symbolic AI, which is a combination of symbolic reasoning and statistical learning. The point of such systems is that they contain transparent and intelligible axioms and rule sets that can be used not only to validate models, but also to explain outputs to humans.

These and similar developments represent technical solutions to regulatory and commercial problems. The current disconnect between technical solutions and policy development, however, poses challenges. Explainability is a multi-disciplinary problem and ought to involve the direct interaction of technical, legal, and policy experts. Broad-based commercialization of AI requires expansion of industry and regulatory forums that include technical components and experts. This requires engagement at the regulatory level for multiple industries and across multiple agencies, not just reliance on national legislation. While engagement is happening, industry will benefit from many more similar efforts across a broader spectrum of society.

Explainability of this kind will also reduce the risk that users will simply prescriptively apply instructions dictated to them by artificial intelligence systems or simply rely on the outputs and decisions of those systems without critically comprehending them. The safest and most responsible uses of artificial intelligence will require human-AI collaboration that necessarily engages the human user’s critical thinking skills. To achieve this, the logical axioms, and reasoning process used by AI systems to make inferences must, even if not a perfect mathematical representation of the machine-learning algorithm, at least be translated into human-intelligible explanations that allow the application of critical reasoning and, ultimately, human judgment.

Misrepresentation claims and explainability

It is easiest to understand this new concept of explainability by linking it to an old legal concept: misrepresentation.

The tort of negligent misrepresentation was recognized in the 1964 case Hedley Byrne. Today, the elements of the cause of action are:

• There must be a duty of care based on a “special relationship” between the representor and the representee;
• The representation in question must be untrue, inaccurate or misleading;
• The representor must have acted negligently in making the misrepresentation;
• The representee must have relied in a reasonable manner on the negligent misrepresentation; and
• The reliance must have been detrimental to the representee in the sense that damages resulted.

Importantly, negligent misrepresentation does not require proof that the defendant engaged in dishonest or fraudulent conduct. Claims that outputs of AI Systems fall within this category are expanding because such systems are known to produce inaccurate outputs that appear to be reliable. Users, of course, bear a certain responsibility for relying on such outputs; however, not every situation is equal and in some cases users may be led to believe that an output is reliable and then act on that output.

Misrepresentation claims that have arisen in Canada and the United States include:

• Businesses held liable for representations made by AI driven chatbots that provide advice to customers while failing to take reasonable care to ensure the chatbot output/advice was accurate. Tribunals have rejected arguments that customers must double-check information provided by chat bots against information available elsewhere in a company’s platform.
• Claims by content creators against platforms alleging that AI systems used to accept or reject algorithms were biased and the basis for the decisions could not be explained;
• Claims against AI fraud audit tools deployed by government to make decisions categorizing recipients of government benefits as engaging in fraud, and issuing related fines on the basis of biased outputs and decisions that could not be explained; and
• Claims against deployers of AI systems that use biometric and other data to assist in making employment decisions alleging the use of the data and inferences drawn from it were biased and could not be explained.

These examples demonstrate that the combination of black box AI inference and user reliance on AI system outputs is creating liability in negligent misrepresentation for deployers of AI systems. The typical defence to such claims is contractual terms and conditions that contain disclosures and disclaimers respecting the AI system and its intended and prohibited uses. However, contractual terms may be insufficient in certain circumstances.

Explainability adds a powerful layer of protection on top of contractual terms and conditions. Explainability not only assists a user understand the context of an output, but actually provides a set of understandable reasons or an understandable context, that allows the user to apply critical thinking and reasoning skills to appreciate what an output is and what it is not, how that output was derived, and where inferences that led to that output may have gone awry.  

If, for example, a customer-facing generative AI chatbot did not simply produce outputs that appeared to be natural language human-like answers to questions, but also made accessible to the user an explanation that this output was derived from an inference model trained on a specific data set for a limited set of purposes, and then provided a basic chain of logic (even if by analogy) to show the user how their input question generated the output, then that user would likely have little basis to argue that they were provided a representation that the natural language output was definitively true and could be relied upon. Of course, if a chat bot is intended to be relied upon for 100 percent accurate outputs but cannot be, then there is likely little utility to it. The explainability principle highlights the issue.

While the chat bot example may not be commercially practical, there are many examples that are. For instance, there are currently tools that allow a business end-user to access a company data warehouse without the assistance of a data analyst and generate outputs based on the user inputs querying the data warehouse for certain information or to produce certain reports based on company data. These systems also contain components that, when clicked on by a user, allow that user to see exactly where the AI tool pulled the data from, the relationship between the data sets used, and what is included (and excluded) in the ultimate output. Such tools are commercially useful and, if the explanation is reliable, greatly mitigate the risk that a user will rely on a misrepresentation or assume an output represents something wholly different from what it actually represents.

These examples highlight the relationship between explainability and commercialization. While generative AI remains an experimental technology in its early days, the issues it raises must be grappled with now. Explainability is a cornerstone principle that any business serious about generative AI must understand and engage.