With the rise of AI, there has been a parallel explosion of new terms and acronyms. One of the latest tendencies is Explainable AI, or XAI for short, and it’s more important than a flash-in-the-pan buzzword.
Explainable AI is a set of methods that allows humans to comprehend and trust the results created by machine learning algorithms. While typical AI models provide an output that leaves the logic behind it shrouded in mystery, in a so-called “black box”, XAI gets down and dirty to open the hood.
While AI emerged, people were so wowed that they didn’t ask many deep questions about its decisions. But now, we need XAI because AI systems are increasingly integrated with areas where the stakes couldn’t be higher, like healthcare, academia, criminal justice, energy management, climate action, and even war.
Accepting an algorithm’s verdict on blind trust is no longer tenable – and in many jurisdictions, it’s no longer legal. Folk need to be sure that an AI isn’t hallucinating, relying on biased data, or making catastrophic errors disguised as confident predictions. Which is tricky, because AI chatbots still aren’t mature or wise enough to admit they’re unsure about the issues at hand.
Example 1: Severe, unchallenged academic judgement
Consider the widely-shared real-world story of a senior student at a private university in New York who lost a $45,000 merit scholarship and faced academic suspension because an automated AI detection tool flagged his 30-page senior thesis as “98% likely AI-generated.”
The precipitously high stakes of this scenario became clear when the student, who claimed to have spent six months writing the thesis manually, offered to walk the board through his complete Google Docs version history – a key-by-key record of his human effort. The board refused to even consider looking, stating their policy was to trust the software’s report, unflinchingly – leaving the student helpless against the verdict, which could be a false-positive.
The story remains unverified and was based on a Reddit post (below) – it may itself be a fake – but there are myriad similar accounts emerging. They can’t all be fiction and they highlight why explainable AI is needed. A single, unexplainable “AI or Human” score is an ominous use of machine learning to dictate a human future. An explainable AI detection system wouldn’t spit out an immutable, final, 98% probability; it would highlight the linguistic patterns, perplexity metrics, or lack of burstiness that triggered the judgement.
And also, the core philosophy of XAI mandates human oversight and the chance, in this case, for redemption. In a responsible XAI framework, an algorithmic flag acts as a trigger for human investigation; contextual, real-world evidence – like a document’s edit history – should override an opaque machine verdict.
Example 2: The health of human & planetary bodies
Imagine an AI system used to analyze X-rays and detect early signs of pneumonia.
A traditional black-box model might analyze the scan and output: 94% probability of pneumonia. If the attending physician disagrees based on their clinical assessment, they are stuck. Do they acquiesce to the machine or trust their own formidable expertise?
An explainable AI model, however, outputs the same 94% probability but also highlights the regions of the X-ray (using saliency maps and such) that led to this idea. It might indicate that it flagged a cluster of fluid in the lower left lobe. The physician can now zone in on that exact spot, evaluate the AI’s reasoning and make the final call. The AI acts as an expert second opinion rather than an absolute authority that dictates the diagnosis.
In our own area of sustainability – the same example can be applied to climate. AI is increasingly deployed to aid the health of the planet by analyzing complex satellite imagery to track carbon sequestration, soil viability, illegal deforestation, and so on. A traditional black-box system might scan a swathe of protected forest and illuminate the operator with a high probability of illegal logging, leaving conservationists to either trust the alert or question its validity. An explainable AI model, much like the medical example, provides backup, like a visual heat map over the topographical data, a shift in canopy density, a sudden drop in soil moisture, or the faint outline of an unmapped logging road – that triggered the warning. One of the brands featured in our roundup of AI for good does just this, with time-lapse animations and series data, showing the changes over time. A human can then verify combined with their own ecological expertise.
Just as with the physician, the AI becomes a partner in protecting the planet, rather than an omniscient oracle that answers to no-one.
Is Explainable AI for people or developers?
Explainable AI is helpful to both but it serves different purposes for each audience:
- For users: XAI acts as a transparency tool. It provides intuitive justifications for automated decisions, so users can understand the AI’s output – as we’ve described in our earlier examples.
- For developers: XAI acts as a debugging / evaluation tool. It helps engineers understand model logic, root-out biases and make sure the system functions fairly before release.
Tools powering transparency
To achieve a desirable level of transparency, developers are relying on new technology that probes model behavior.
It’s still early days but a new standard in this space is Google’s What-If Tool (WIT). Designed as an interactive visual interface, the What-If Tool allows data scientists, ML practitioners, and non-experts to ask “what if?” without writing complex code. You can manually tweak a data point (e.g., changing a loan applicant’s age or income) and see how the model’s prediction changes, instantly. This makes it easier to uncover hidden biases before a flawed algorithm is unleashed on the masses to wreak havoc (or at least frustration) with unfair judgements from its walled garden in the cloud.
6 key benefits of explainable AI
In summary, here are the key benefits of explainable AI – for both users and developers.
- Trust: AI can be perceived as a sinister, inscrutable, threatening “black box”. But if people can understand how it makes decisions, it loses that mysterious edge and can be adopted in the spirit of a collaborative tool.
- Compliance: With frameworks like the EU AI Act beginning to mandate transparency for high-risk systems, XAI provides the necessary evidence trails that prove compliance and avoid legal penalties.
- Bias: By exposing the weight given to different data points, XAI makes it possible to spot when a model is discriminating against certain demographics, allowing developers to correct these structural -isms.
- Performance: When data scientists can see where and why a model is making illogical leaps, they can fine-tune the training data and architecture more efficiently than when making educated guesses.
- Recourse: If an AI denies a service (like a bank loan), XAI provides a specific reason (e.g., the debt-to-income ratio being too high), giving the user a negotiable route to improve the result next time rather than a brick-wall rejection.
- Sustainability: While explanations require an upfront energy cost, XAI acts as an environmental audit for algorithms. By revealing which data points matter, it allows developers to discard redundant data and optimize for efficient AI systems that use less energy. This is an important point as the environmental impact of AI is getting really disturbing.
When we fail to prioritize these benefits, we inch closer to a reality we’ve long been warned about in films, books, and from prescient thinkers. We’ll end on that…
An unavoidable cliché: Dystopia
A future where AI doesn’t need to explain itself while making judgements that affect the lives of humans is intensely dystopian. Such a scenario could make for a thrilling sci-fi novel, written as a warning, but a rather frightening reality. Strangely, the idea is, little by little, starting to shed the reassuring patina of fantasy. It is moving closer.
We must hope that more transparency arrives soon. Earlier, we didn’t mention the example of Anthropic, who famously refused to give the US military unfettered access to its technology, where explainability could’ve been compromised against a backdrop of life or death – drones zipping overhead. That’s a bright light because what’s needed, as well as appropriate technology, is more humans with ethics and standards; less greed. After all, a lack of transparency also means more efficiency, higher speed, and larger profits.
Ethical attributes are the allies of Explainability. We do hope that more people start to feel and then show them. In the meantime, raising awareness is our goal and we hope this article has been enlightening.
Leave a Reply