The “interpretation methods” seek to unravel how machine learning models make predictions, but the researchers say it should be done with caution.
About a decade ago, deep learning models started to achieve superhuman results in all kinds of tasks, from beating world champions board game players arrive surpassing doctors at diagnosis of breast cancer.
The strong deep learning model usually based on artificial neural network, was first proposed in the 1940s and has become a popular type of machine learning. Computers learn to process data using layers of interconnected nodes, or neurons, that mimic the human brain.
As the field of machine learning grows, so do artificial neural networks.
Today’s deep learning models typically consist of millions or billions of nodes connected together in multiple layers trained to perform detection or classification tasks using large amounts of data. But because the models are so complex, even the researchers who designed them don’t fully understand how they work. This makes it difficult to tell if they are working correctly.
For example, maybe a model designed to help doctors diagnose patients correctly predicts that a skin lesion is cancerous, but it did so by focusing on an unrelated sign. usually occurs when there is cancerous tissue in the image, rather than on the tumor itself. This is called pseudo-correlation. The model makes the right prediction, but it does so for the wrong reason. In a realistic clinical setting where the sign does not appear on cancer-positive imaging, it can lead to missed diagnosis.
With so much uncertainty surrounding these so-called “black box” models, how can one shed light on what is happening inside the box?
This puzzle has led to a new and rapidly growing field of research in which researchers develop and test explanatory methods (also called interpretive approaches) in an attempt to unravel how black box machine learning models make predictions.
What are the demonstration methods?
At their most basic level, interpretation methods can be global or local. Local interpretation focuses on explaining how the model performs a particular prediction, while global interpretation seeks to describe the overall behavior of the entire model. This is usually done by developing a separate, simpler (and hopefully understandable) model that mimics the larger black box model.
But because deep learning models behave in fundamentally complex and non-linear ways, developing an effective global explanatory model is a particular challenge. This has led researchers to shift much of their focus to local explanatory methods instead, said Yilun Zhou, a PhD student in the Interactive Robotics Group of the Computer Science and Intelligence Laboratory. artificial intelligence (CSAIL), who studies models, algorithms, and assessments in interpretable, interpretable machine learning.
The most common types of local interpretation methods are divided into three broad categories.
The first and most widely used type of interpretation method is called feature allocation. The feature attribution method indicates which features are most important when the model makes a particular decision.
Features are input variables fed to a machine learning model and used in its prediction. When the data is tabular, features are drawn from the columns in the data set (they are transformed using various techniques so that the model can process the raw data). On the other hand, for image processing tasks, each pixel in the image is a feature. For example, if a model predicts that an X-ray image shows cancer, the feature attribution method will highlight the pixels in that particular X-ray that are most important to the model’s prediction.
Basically, feature attribution methods show what the model pays the most attention to when making predictions.
“Using this feature attribution interpretation, you can test to see if pseudo-correlation is a concern. For example, it will show if pixels in a watermark are marked or if pixels in a tumor are indeed marked,” Zhou said.
A second type of explanatory method is called the counterfactual method. Given an input and a prediction of the model, these methods show how to change that input so that it falls into a different class. For example, if a machine learning model predicts that a borrower will be denied a loan, the reverse explanation of reality shows what factors need to change in order for their application to be approved. Perhaps her credit score or income, both characteristics used in the model’s predictions, need to be higher for her to be approved.
“The good thing about this method of interpretation is that it tells you exactly how you need to change the input to flip a decision, which may have practical uses. For someone who is applying for a mortgage and not getting it, this explanation will tell them what they need to do to achieve the desired outcome,” he said.
The third type of explanatory method is called pattern significance interpretation. Unlike other methods, this method requires access to the data used to train the model.
An explanation of sample importance will indicate which training pattern a model relies on most when making a particular prediction; Ideally, this is the most similar pattern to the input data. This type of explanation is especially useful if an observer finds a prediction that seems irrational. There may have been an import error that affected a particular sample used to train the model. With this knowledge, one can repair that sample and retrain the model to improve its accuracy.
How are narrative methods used?
One motivation for developing these interpretations was to perform QA and model debugging. For example, with a better understanding of how features impact model decisions, one can determine that a model is performing incorrectly and intervene to troubleshoot or toss the model. out and start over.
Another, more recent area of research is exploring the use of machine learning models to discover scientific patterns that have not been discovered before by humans. For example, a cancer diagnostic model that works better than clinicians may be faulty, or it may actually be finding some hidden pattern in the X-ray image that represents a disease pathway. early symptoms of cancer that are unknown to human doctors, or are believed to be, Zhou said.
However, it is still very early for that area of research.
Warning
Marzyeh Ghassemi says: While explanatory methods can sometimes be helpful to machine learning practitioners when they are trying to catch errors in their model or understand the inner workings of a system, users are Finally, caution should be exercised when trying to use them in practice. , an assistant professor and head of the Healthy ML Group at CSAIL.
As machine learning has been applied in more fields, from healthcare to education, explanatory methods are being used to help decision makers better understand the model’s predictions so they know when to trust the model and use its guidance in practice. But Ghassemi warn against using these methods in that way.
“We found that the explanations led people, both experts and non-experts, to over-confident the capabilities or advice of a particular recommendation system. I think it’s very important that humans don’t shut down that internal circuitry, ‘let me question the advice that I
she speaks.
Scientists know explanations that make people overconfident based on other recent work, she added, citing several recently learn by Microsoft researchers.
Far from being a silver bullet, explanatory methods have their share of problems. First, recent research by Ghassemi has shown that explanatory methods can maintain prejudice and lead to worse outcomes for people in disadvantaged groups.
Another pitfall of interpretation methods is that it is often impossible to know if the interpretation method is correct in the first place. One would need to compare the explanations with the actual model, but since users don’t know how the model works, this is circular logic, Zhou said.
He and other researchers are working to improve their interpretation methods so that they are more faithful to the predictions of the real model, but Zhou warns that even the best explanation should be take a grain of salt.
“Also, people often see these models as human-like decision makers, and we are prone to over-generalizing. We need to reassure people and hold them back to really make sure that the understanding of the general model they build from these local interpretations is balanced,” he added.
By Zhou most recent research find a way to do it.
What is the next method of machine learning interpretation?
Rather than focusing on providing explanations, Ghassemi argues that the research community needs to put more effort into studying how information is presented to decision-makers so that they make sense of it and need it. There are more regulations to ensure machine learning models are used responsibly in practice. Better explanatory methods are not the answer.
“I’m thrilled to see that there’s so much recognition, even in the industry, that we can’t just take this information and create a nice dashboard and assume that people will do better with this. there. You need measurable improvements in action, and I hope that leads to real principles of improving the way we display information in these deeply technical areas, like medicine,” she said.
And in addition to the new work focused on improving interpretations, Zhou hopes to see more research related to interpretation methods for specific use cases, such as debugging models, exploring scientific investigation, fair audit and safety assurance. By identifying the detailed characteristics of the methods of explanation and the requirements of different use cases, researchers can establish a theory that matches explanations to specific situations. , which can help overcome some of the pitfalls of using them in real-world scenarios.
Written by Adam Zewe
Source: Massachusetts Institute of Technology
