Medical AI: addressing the validation gap
Medical AI: addressing the validation gap
School of Computing Science
Date: Wednesday 22 March 2023
Time: 14:00 - 15:00
Venue: Online: https://uofglasgow.zoom.us/meeting/register/tZwrf-ygqjguE9NlUx9-uLqR0IEutLzPDe0K
Category: Public lectures, Academic events, Student events, Alumni events, Courses
Speaker: Dr. Gael Varoquaux
Website: gael-varoquaux.info/about.html
Abstract:
Machine-learning, which can learn to predict given labeled data, bares many promises for medical applications. And yet, experience shows that predictors that looked promising most often fail to bring the expected medical benefits. One reason is that they are evaluated detached from actual usage and medical outcomes.
And yet, test runing predictive models on actual medical decisions can be costly and dangerous. How do we bridge the gap? By improving machine-learning model evaluation. First, the metrics used to measure prediction error must capture as well as possible the cost-benefit tradeoffs of the final usage. Second, the evaluation procedure must really put models to the test: on a representative data sample, and accounting for uncertainty in model evaluation. I will discuss advanced topic on these questions.
For medical applications, predictions should come with associated confidence. It is important to evaluate these confidence with adequate metrics. Here, the difficulty is to control individual probabilities, as each individual is observed only once. I will explain a procedure to measure how far a predictor is from outputing the ideal individual probabilities, due to intrinsic uncertainty [1].
Predictors can be used to reason about possible interventions: for a given individual, what is the potential outcome of an intervention versus no intervention? However, the corresponding inferences require a particular type of control on the error of the predictors [2].
Last but not least, a numerical experiment to benchmark predictors comes with arbritrary sources of variation. Understanding and accounting for this uncontroled variance is important to make well-grounded decisions on which predictive model to use. This is possible with simple procedures [3].
[1] Beyond calibration: estimating the grouping loss of modern neural networks
Alexandre Perez-Lebel, Marine Le Morvan, Gaël Varoquaux
ICLR 2023 – The Eleventh International Conference on Learning Representations, May 2023, Kigali, Rwanda
https://hal.science/hal-03829870v3
[2] How to select predictive models for causal inference?
Matthieu Doutreligne, Gaël Varoquaux
2023
https://hal.science/hal-03946902
[3] Accounting for variance in machine learning benchmarks.
Xavier Bouthillier, et al.
Proceedings of Machine Learning and Systems 3 (2021): 747-769.
https://proceedings.mlsys.org/paper/2021/hash/cfecdb276f634854f3ef915e2e980c31-Abstract.html
Biography:
Gael is a research director at the National Institute for Research in Digital Science and Technology (INRIA) at France. He is also the team leader of Soda - Computational and Mathematical Methods to understand health and society with data (https://team.inria.fr/soda/). His research interests encompasses three areas:
- Machine learning and public health, which involves analytics on health databases for personalized medicine and treatment development, biomedical natural language processing and information extration and causal inference.
- Democratizing machine learning which encompasses machine learning on dirty data Missing data in machine learning, machine-learning model evaluation and learning on relational databases.
- Machine learning for mental health, cognition, and brain activity, which encompasses learning models of brain function and its pathologies from brain imaging, biomarkers of mental traits and disorders which encompasses resting-state and functional connectivity Encoding and decoding models of cognition.
He is also the director of scikit-learn operations at Inria foundation and core contributor of several open source projects in scientific computing with python.
Registration is required: https://uofglasgow.zoom.us/meeting/register/tZwrf-ygqjguE9NlUx9-uLqR0IEutLzPDe0K