With the rise of AI, there has been a parallel explosion of new terms and acronyms. One of the latest tendencies is Explainable AI, or XAI for short, and it’s more important than a flash-in-the-pan buzzword.
Explainable AI is a set of methods that allows humans to comprehend and trust the results created by machine learning algorithms. While typical AI models provide an output that leaves the logic behind it shrouded in mystery, in a so-called “black box”, XAI gets down and dirty to open the hood.
While AI emerged, people were so wowed that they didn’t ask many deep questions about its decisions. But now, we need XAI because AI systems are increasingly integrated with areas where the stakes couldn’t be higher, like healthcare, academia, criminal justice, energy management, climate action, and even war.
Accepting an algorithm’s verdict on blind trust is no longer tenable – and in many jurisdictions, it’s no longer legal. Folk need to be sure that an AI isn’t hallucinating, relying on biased data, or making catastrophic errors disguised as confident predictions. Which is tricky, because AI chatbots still aren’t wise enough to admit they’re unsure about the issues at hand.
Example 1: Severe, unchallenged academic judgement
Consider the widely-shared real-world story of a senior student at a private university in New York who lost a $45,000 merit scholarship and faced academic suspension because an automated AI detection tool flagged his 30-page senior thesis as “98% likely AI-generated.”
The precipitously high stakes of this scenario became clear when the student, who claimed to have spent six months writing the thesis manually, offered to walk the board through his complete Google Docs version history – a key-by-key record of his human effort. The board refused to even consider looking, stating their policy was to trust the software’s report, unflinchingly – leaving the student helpless against the verdict, which could’ve been a false-positive.
The story remains unverified and was based on a Reddit post (below) – it may itself be a fake – but there are myriad similar accounts emerging. They can’t all be fiction and they highlight why explainable AI is needed. A single, unexplainable “AI or Human” score is an ominous use of machine learning to dictate a human future. An explainable AI detection system wouldn’t spit out an immutable, final, 98% probability; it would highlight the linguistic patterns, perplexity metrics, or lack of a characteristic like burstiness that triggered the judgement.
And also, the core philosophy of XAI mandates human oversight and the chance, in this case, for redemption. In a responsible XAI framework, an algorithmic flag acts as a trigger for human investigation; contextual, real-world evidence – like a document’s edit history – should override a machine verdict.
Example 2: The inscrutable algorithms of the gig economy
Another real-world case is the plight of food delivery riders navigating our cities for platforms like Deliveroo, Uber Eats, and Just Eat. In the picturesque streets of Edinburgh, a group of workers say their take-home pay has plummeted by up to half over the last few years, despite them working the same hours and delivering the same orders.
Behind this worsening of conditions, they claim, are incomprehensible algorithms that oversee dynamic pricing and job allocation. These AI algorithms process real-time supply and demand to calculate a fluctuating delivery fee. If a rider’s average earnings inexplicably drop, or if they are penalized by the system for rejecting low-paying stacked orders, they are left in the dark. And this automation can have other disastrous consequences: one rider was allegedly locked out of his livelihood entirely when the app’s facial recognition system failed to recognize him after he suffered a black eye, leaving him helpless against the machine’s verdict with no access to human support.
An explainable AI model would bring some fairness and clarity to this crushing dynamic. Instead of dictating an immutable £3.42 fee for a delivery, it would reveal the variables – such as localized demand, distance, job history, or restaurant wait times – that led to the price. If an account were flagged or a facial recognition check failed, an explainable system would identify the issue (like a feature mismatch) and trigger a human investigation. There would be no brick-wall rejection.
Just as with the academic board, XAI in the gig economy is a way to bring about transparent tools for efficiency, rather than build amoral data-center deities that cannot be reached or appealed to.
Meanwhile, in Edinburgh, where the streets are beautiful and the algorithms inexplicable, the workers in question aim to enlist the help of academics from the University of Edinburgh and St Andrews – two of the world’s finest – to figure out what on earth is happening to them and the deliveries inside their own insulated boxes. Something they’ve been powerless to achieve so far.
Example 3: The health of human & planetary bodies
Imagine an AI system used to analyze X-rays and detect early signs of pneumonia.
A traditional black-box model might analyze the scan and output: 94% probability of pneumonia. If the attending physician disagrees based on their clinical assessment, they are stuck. Do they acquiesce or trust their own formidable expertise?
An explainable AI model, however, would output the same 94% probability but also highlight the regions of the X-ray (using saliency maps and such) that led to this idea. It might indicate that it flagged a cluster of fluid in the lower left lobe. The physician could now zone in on that exact spot, evaluate the AI’s reasoning and make the final call. The AI acts as an expert second opinion rather than an absolute authority that dictates the diagnosis.
In our own area of sustainability – the same example can be applied to climate. AI is increasingly applied to aid the health of the planet by analyzing complex satellite imagery to track carbon sequestration, soil viability, illegal deforestation, and so on. A traditional black-box system might scan a swathe of protected forest and illuminate the operator with a high probability of illegal logging, leaving conservationists to either trust the alert or question its validity. An explainable AI model, much like the medical example, provides backup, like a visual heat map over the topographical data, a shift in canopy density, a sudden drop in soil moisture, or the faint outline of an unmapped logging road – that triggered the warning. One of the brands featured in our roundup of AI for good does just this, with time-lapse animations and series data, showing the changes over time. A human can then verify combined with their own ecological expertise.
Just as with the physician, the AI becomes a partner in protecting the planet, rather than an omniscient oracle that answers to no-one.
Is Explainable AI for people or developers?
Explainable AI is helpful to both but it serves different purposes for each audience:
- For everyday people: XAI acts as a transparency tool. It provides intuitive justifications for automated decisions, so users can understand the AI’s output – as we’ve described in our earlier examples. When life-changing decisions are made, it helps bring AI to account.
- For developers: XAI acts as a debugging / evaluation tool. It helps engineers understand model logic, root-out biases and make sure the system functions fairly before release. The process is a way for developers step inside the complex ‘minds’ of new models and their decision making.
Tools powering transparency
To achieve a desirable level of transparency, developers are relying on new technology that probes model behavior.
It’s still early days but a new standard in the developer side of XAI is Google’s What-If Tool (WIT). Designed as an interactive visual interface, the What-If Tool allows data scientists, ML practitioners, and non-experts to ask “what if?” without writing complex code. You can manually tweak a data point (e.g., changing a loan applicant’s age or income) and see how the model’s prediction changes, instantly. This makes it easier to uncover hidden biases before a flawed algorithm is unleashed on the masses to wreak havoc (or at least frustration) with unfair judgements from its walled garden in the cloud.
On the everyday side, AI detection is an area where demand is soaring but transparency is lagging. CopyLeaks’ AI detector – developed by one of the fastest growing companies in the US – demystifies its verdicts with ‘AI Logic’, revealing the why behind the judgements of AI or not-AI. This logic provides detailed evidence, such as source matches and statistical frequencies – which flag the predictable word choices and patterns that give AI away. It’s a tool that’s way more transparent than most AI detection tools. It would’ve been helpful in the case of the student we mentioned earlier, if as innocent as the story suggests.
6 key benefits of explainable AI
In summary, here are the key benefits of explainable AI – for both users and developers.
- Trust: AI can be perceived as a sinister, inscrutable, threatening “black box”. But if people can understand how it makes decisions, it loses that mysterious, menacing edge and can be welcomed in the spirit of a collaborative technology.
- Compliance: With frameworks like the EU AI Act beginning to mandate transparency for high-risk systems, XAI provides the necessary evidence trails that prove compliance and avoid legal penalties.
- Bias: By exposing the weight given to different data points, XAI makes it possible to spot when a model is discriminating against certain demographics, allowing developers to correct these structural -isms.
- Performance: When data scientists can see where and why a model is making illogical leaps, they can fine-tune the training data and architecture more efficiently than when making educated guesses.
- Recourse: If an AI denies a service (like a bank loan), XAI provides a specific reason (e.g., the debt-to-income ratio being too high), giving the affected human a negotiable route to improve the result next time rather than a rejection that can’t be overcome.
- Sustainability: While explanations require an upfront energy cost, XAI acts as an environmental audit for algorithms. By revealing which data points matter, it allows developers to discard redundant data and optimize for efficient AI systems that use less energy. This is an important point as the environmental impact of AI is getting really disturbing.
When we fail to prioritize these benefits, we inch closer to a reality we’ve long been warned about in films, books, and from prescient thinkers. We’ll end on that…
An unavoidable cliché: Dystopia
A future where AI doesn’t need to explain itself while making judgements that affect the lives of humans is intensely dystopian. Such a scenario could make for a thrilling sci-fi novel, written as a warning, but a rather frightening reality. Strangely, the idea is, little by little, starting to shed the reassuring patina of fantasy. It is moving closer.
We must hope that more transparency arrives soon. Earlier, we didn’t mention the example of Anthropic, who famously refused to give the US military unconditional access to its technology, where explainability could’ve been compromised against a backdrop of life or death – drones zipping overhead. That’s a bright light because what’s needed, as well as appropriate technology, is more humans with ethics and standards; less greed. After all, a lack of transparency also means more efficiency, higher speed, and larger profits.
Ethical traits in humans are the allies of AI Explainability. We do hope that more people start to feel and then show them. In the meantime, raising awareness is our goal and we hope this article has been enlightening.
Leave a Reply