Artificial intelligence is gaining traction in healthcare, with many hospitals and health systems implementing the technology — often in administrative settings — with great success.
But success with AI in healthcare – especially on the clinical side – cannot be achieved without addressing growing concerns about the transparency and explainability of models.
In a field where decisions can be life-or-death, being able to understand and trust AI decisions is not just a technical need – It is morally imperative.
Neeraj Mainkar is vice president of software engineering and advanced technology at Proprio, which develops immersive tools for surgeons. He has significant expertise in applying algorithms in healthcare. Healthcare IT News spoke with him to discuss explainability, the need to ensure patient safety and trust, identifying errors, regulatory compliance, and ethical standards in AI.
Q. What does explainability mean in the field of artificial intelligence?
ONE. Explainability refers to the ability to understand and clearly express how an AI model came to a particular decision. In simpler AI models, such as decision trees, this process is relatively straightforward because the decision paths can be easily followed and interpreted.
However, as we move into the realm of complex deep learning models, which involve multiple layers and complex neural networks, the challenge of understanding the decision-making process becomes significantly more difficult.
Deep learning models operate with large numbers of parameters and complex architectures, making it nearly impossible to directly trace their decision paths. Reverse engineering these models or examining specific issues in the code is extremely difficult.
When a prediction does not match expectations, it is difficult to pinpoint the exact reason for this discrepancy due to the complexity of the model. This lack of transparency means that even the creators of these models may have difficulty fully explaining their behavior or output.
Opacity of Complex AI systems pose significant challenges, especially in areas like healthcare where understanding the reasoning behind a decision is critical. As AI continues to become more deeply integrated into our lives, the need for explainable AI is growing. Explainable AI aims to make AI models more interpretable and transparent, ensuring their decision-making processes can be understood and trusted.
Q. What are the technical and ethical implications of explainable AI?
ONE. The effort to explain has both technical and ethical implications to consider. Technically, simplifying models to increase explainability can reduce performance, but it can also help AI engineers debug and improve algorithms by giving them a clear understanding of where their outputs come from.
Ethically, explainability helps identify bias in AI models and promotes fairness in treatment, eliminating discrimination against smaller, underrepresented groups. Explainable AI also ensures end users understand how decisions are made while still protecting sensitive information in compliance with HIPAA.
Q. Please discuss the error identification regarding interpretability.
ONE. Explainability is a critical component to effectively identifying and correcting errors in AI systems. The ability to understand and interpret how an AI model reaches its decisions or outputs is essential to effectively identifying and correcting errors.
By tracing decision paths, we can identify where the model may be going wrong, allowing us to understand the “why” behind an incorrect prediction. This understanding is critical to making the necessary adjustments to improve the model.
Continuously improve AI models depends heavily on understanding their failures. In healthcare, where patient safety is paramount, the ability to debug and refine models quickly and accurately is crucial.
Q. Please elaborate on the regulatory compliance regarding explainability.
ONE. Healthcare is a highly regulated industry with strict standards and guidelines that AI systems must meet to ensure safety, effectiveness, and ethical use. Explainability is critical to achieving compliance, as it addresses several key requirements, including:
- Transparency. Explainability ensures that every decision made by AI can be traced and understood. This transparency is essential to maintain trust and ensure AI systems operate within ethical and legal boundaries.
- Authenticate. Explainable AI helps demonstrate that models have been thoroughly tested and validated to perform as expected in a variety of scenarios.
- Minimize bias. Explainability enables the identification and mitigation of biased decision-making models, ensuring models do not unfairly disadvantage any particular group.
As AI continues to evolve, an emphasis on explainability will continue to be an important aspect of the regulatory framework, ensuring these advanced technologies are used responsibly and effectively in healthcare.
Q. So how do ethical standards relate to explainability?
ONE. Ethical standards play a fundamental role in the development and deployment of responsible AI systems, especially in sensitive and high-risk areas such as healthcare. Explainability is inherent in these ethical standards, ensuring that AI systems operate transparently, fairly, and responsibly, in line with core ethical principles in healthcare.
Responsible AI means operating within ethical boundaries. Promoting enhanced explainability in AI enhances trust and reliability, ensuring that AI decisions are transparent, defensible, and ultimately beneficial to patient care. Ethical standards guide responsible disclosure, protect user privacy, uphold regulatory requirements such as HIPAA, and promote public trust in AI systems.
Follow Bill’s HIT coverage on LinkedIn: Bill Siwicki
Email him: [email protected]
Healthcare IT News is a publication of HIMSS Media.
