Counterfactuals are ideas about an alternative outcome of an event. These are subjunctive sentences of the type “if …. then …”. How can we relate this to AI systems?
Let’s say a person named Sarah applies for a loan. The bank asks him to provide some information such as his marital status, debt level, income, savings, home address and age. The bank then sends this information to an artificial intelligence system that returns a prediction about whether Sarah will repay the loan. However, neither Sarah nor the bank employees know why the system gives such a score. In order to make the system more transparent, it should be tried to explain how the model works.
One approach is to examine the inputs and outputs of the pattern-seeking system. This is where counterfactuals come into play, statements about what would have happened if things had gone differently. In the context of artificial intelligence, this means thinking about what the output will be if the system receives different inputs.
Suppose the bank sends different, modified information about Sarah to the AI system. Based on this, the bank calculates that the smallest change required to achieve a positive result is an increase in revenue. The negative result can be explained as follows: Sarah’s loan was denied because her income was too low.
It sounds good, but researchers argue that the counterfactual approach is inadequate. When we look at changes in inputs and how they translate into outputs, we learn about correlations. But as the old saying goes, correlation is not causation. Therefore, it may be a mistake for the bank to tell Sarah that she was not given a loan because her income was too low. All he can say for sure is that income and credit score are correlated. Sarah will remain without an explanation.
Most likely, you need to use a different approach to create “explainable AI”.
Source: Ferra
