A team of researchers from the University of Geneva (UNIGE), University Hospitals of Geneva (HUG), and the National University of Singapore (NUS) has created a pioneering approach to evaluating AI interpretation methods. The goal is to uncover the basis for AI decision making and identify potential biases.

A team of researchers from the University of Geneva (UNIGE), University of Geneva Hospitals (HUG) and the National University of Singapore (NUS) has developed a new approach to evaluate the interpretation of artificial intelligence (AI) technologies. This breakthrough paves the way for greater transparency and trust in AI-powered diagnostic and predictive tools.

The new method sheds light on the mysterious workings of so-called “black box” AI algorithms, helping users understand what influences AI-generated results and whether the results can be trusted. This is especially important in scenarios that have a significant impact on human health and well-being, such as the use of artificial intelligence in medical applications.

The research is particularly relevant in the context of the forthcoming European Union Artificial Intelligence Law, which aims to regulate the development and use of artificial intelligence in the EU. The findings were recently published in a journal Nature Machine Intelligence.

Time series data representing the evolution of knowledge over time is ubiquitous: in medicine, for example, when recording heart activity with an electrocardiogram (ECG); earthquake studies; monitoring of weather conditions; or in the economy to monitor financial markets. This data can be modeled by artificial intelligence technologies to create diagnostic or predictive tools.

The development of artificial intelligence, and especially deep learning, which consists of teaching a machine to interpret this huge volume of data and learn useful patterns, is paving the way to increasingly accurate tools for diagnosis and prediction. However, without understanding how AI algorithms work or what influences their results, the “black box” nature of AI technology raises important questions about reliability.

“The way these algorithms work is incomprehensible to say the least,” says Professor Christian Lovis, head of the Radiology and Medical Informatics Division at UNIGE School of Medicine and Head of Health Informatics at HUG.

“Of course, the risks, especially financial risks, are extremely high. But how can we trust a machine without understanding the basics of its logic? These questions are especially important in industries like medicine, where AI-based solutions can impact people’s health and even lives; and finance, where they can result in a huge loss of capital.

Interpretation techniques aim to answer these questions by deciphering why and how AI makes a particular decision and the reasons behind it. Director of MathEXLab, Faculty of Design and Engineering, National University of Singapore, Assoc. who ran the business.

“However, current interpretation methods, which are widely used in practical applications and industrial workflows, yield markedly different results when applied to the same task. This raises an important question: Which interpretation method is correct, given that there must be a unique correct answer? Therefore, evaluation of interpretation methods becomes as important as interpretability itself.”

distinguish the important from the unimportant

Discriminatory data is critical to the development of interpretable AI technologies. For example, when AI analyzes an image, it focuses on a few characteristic attributes.

Hugues Turbe, a PhD student in Professor Lowis’ lab and first author of the study, explains: “Artificial intelligence can, for example, distinguish the image of a dog from the image of a cat. The same principle applies to the analysis of time series: the machine must be able to choose the elements on which it will base its judgment—for example, peaks that are more pronounced than others. With EKG signals, this means matching signals from different electrodes to evaluate possible mismatches that could be indicative of a particular heart condition.

It is not easy to choose from all the interpretation methods available for a particular purpose. Different methods of interpreting AI often produce very different results, even when applied to the same dataset and task.

To address this issue, researchers developed two new evaluation methods to help understand how AI makes decisions: one to identify the most relevant parts of the signal, and the other to evaluate their relative importance for the final prediction. To assess interpretability, they withheld some of the data to see if it was relevant to AI decision making.

However, this approach sometimes caused errors in the results. To fix this, they trained the AI ​​on an expanded dataset containing confidential data that helps keep the data stable and accurate. The team then created two ways to measure how well the interpretation methods worked, showing whether the AI ​​was using the right data to make decisions and whether all data was being evaluated fairly. “Overall, our method aims to evaluate a model that will actually be used in the operational domain, thereby ensuring its reliability,” explains Hugues Turbet.

To further their research, the team developed a synthetic dataset that they made available to the scientific community to easily evaluate any new AI aimed at interpreting time series.

The future of medical programs

Next, the team plans to test their method in clinical settings where fears about AI are common. The head of the machine learning group in Professor Lovis’s department and the study’s second author, Dr. “Building confidence in AI assessment is an important step towards its application in clinical settings,” explains Mina Belogrlic. “Our research focuses on AI assessment based on time series, but the same methodology can be applied to AI based on other modalities such as images or text used in medicine.”