School of Computing Science

WP1: Closed-loop interaction with probabilistic models

Evdoxia Taka, Sebastian Stein, John H. Williamson

Data science as practiced currently is an open-loop, statistician-controlled process: from a static dataset, models are learned and a subset of evaluation data is presented via fixed numerical and visual representations. There is limited scope for users to probe:

• whether a representation accurately captures domain knowledge;
• sensiivity to specific data points;
• uncertainty in predictions;
• or alternative causal explanations of the same observations.
  

We focus on Bayesian probabilistic inference, where analysis results in probability distributions over parameters (and potentially predicted outcomes). There is a narrow interface channel between users and the results of an inference process. We view the process as a closed-loop alignment of belief between users and inference results.

Why is this a closed-loop problem?

The closed-loop is between internal human belief states and probabilistic belief states implied by a model and dataset; the underlying hypothesis being that interactive control over the implications of a probabilistic model is required to effectively communicate it.  Closing this loop becomes increasingly important as sophisticated models form the basis for human decision-making. We explore methods that support closed-loop interaction between user groups (statisticians, experimenters, domain experts and end users) that explicitly model and promote interaction with uncertainty.

This requires new processes and tools to turn static data into interactive systems; it requires  modeling of predicted user interest; and latency-precision tradeoffs to maintain a stable flow of information. We view the problem as one of designing a closed-loop for performing synthetic experiments on the results of inference, and supporting users’ in making optimal experiments on belief distributions.

We focused on integrating interactive tools with standard MCMC-based probabilistic programming languages, with the vision of automatically generating interactive visualisations from a probabilistic program applied to a dataset.

WP1 Developments

We view the problem as one of alignment of human beliefs (mental models) and probabilistic insights (probability distributions, or samples from them). Our assumed inference process is that of an MCMC engine taking (priors, model, data) and producing (posterior, posterior_predictive) samples. The research problem is to take these components and “compile” them into a closed-loop system to align users’ beliefs with the inference output. 

We have pursued this through the development of interactive systems to allow users to perform efficient experiments on the implications of a probabilistic inference engine. Latterly, we developed tools for causal exploration where these real-time synthetic experiments become explicit interventions in multiple structurally distinct probabilistic simulators fitted to a common dataset. In parallel, we have embedded interactive counterfactual analysis tools in a sequential human-controlled optimisation loop with our industrial partner Aegean, where probabilistic models suggest interventions via Bayesian optimisation.

Theoretical developments

1. Closed-loop conditioning. The formulation and classification of visualisation and interaction techniques for interactive conditioning of probability distributions. WP1 focused on interactive conditioning on one or more marginal distributions as the primary method for closed-loop exploration of probabilistic models. 
2. Evaluation of probabilistic closed-loop visualisations. Development of robust evaluation protocols for interactive visualisations of probabilistic systems, including measures of rationality under utility functions as measured through simulated lotteries, and comprehension as measured by divergence between true and mental models.
3. Simultaneous interactive causal interventions. Models of interactive exploration of a collection of competitive causal Bayesian models, using simultaneous synthetic interventions on probabilistic simulators trained on identical data.

Practical developments

1. A visualisation platform for communication of Bayesian models with multiple univariate outcomes, including interactive conditioning, hypothetical outcome plots and smooth animated tours of conditional distributions.
2. An interactive tool for comparative exploration of multiple causal explanations, based on MCMC sampling from multiple model structures trained on the same data, allowing a wide range of interactive interventions inside the model.
3. A use case for interactive counterfactual analysis in a closed-loop human-controlled optimisation loop.

Each of these works with a standard probabilistic programming language (pymc3) and operates on a posterior trace and a skeleton model description.

WP2: Closed-loop Data science and Pulmonary hypertension

Recent advances in physiological modelling allow the pulmonary blood pressure to be predicted from the vasculature geometry and blood flow times series, which can be measured non-invasively with computed tomography angiography (CTA) or magnetic resonance angiography (MRA), respectively. The biophysical model depends on various boundary conditions and physiological parameters, most notably the blood vessel stiffness, which can be estimated with computational inference procedures.

 

 

Figure 1: Fluid dynamics model of pulmonary blood circulation.

 

Figure 1 provides a graphical illustration of the model. We are given a blood vessel network, obtained e.g. from CTA or MRA. Blood flow (q) and pressure (p) in this network are modelled as a function of location(x) and time (t) with partial differential equations, which have been derived under certain physical assumptions (that the fluid is viscous, homogeneous and incompressible, and that the flow is axisymmetric, laminar and without swirls). The blood flow at the main pulmonary artery (at the top of the figure) is measured with ultrasound. The effect of the small blood vessels downstream, which are too small to be reliably extracted with CTA or MRA, are approximated with electronic circuits composed of two resistances (R₁ and R₂) and one capacitance (C) (so-called Windkessel elements),which determine the frequency (ω) dependent impedance Z_wk(ω). The model depends on various material parameters (blood density ρ and viscosity ν), which can be directly measured, and various geometry parameters (boundary layer thickness δ, blood vessel radius r, cross section area A and reference area A₀), which can be measured with CTA or MRA. However, the clinically critical parameter is the blood vessel stiffness f, which can not be measured in vivo and has to be inferred, along with the Windkessel parameters, from the blood flow. 

 

Figure 2: Schematic representation of our physiological model of pulmonary hypertension and how it is affected by closed-loop effects following a clinical intervention.

 

Figure 2 provides a schematic illustration of the way this model would be used as part of a clinical decision support system. Given the geometry of the vasculature, most notably the blood vessel diameters (measured with CTA or MRA), the blood flow (measured with ultrasound) and various boundary conditions (obtained from statistical inference), the model allows the prediction of the pulmonary blood pressure and the blood vessel stiffness (with state-of-the-art statistical inference techniques based on MCMC).

WP2

Iadh Ounis, Craig Macdonald, Amir Jadidinejad, Sean Macaveny

Recommender systems are pervasive in daily life - such as in video on demand services exemplified by Netflix, shopping sites such as Amazon or search systems that recommend the best content for specific users. There are a plethora of machine learned models that can infer the latent preferences of users, and the properties of items, and use these to predict the next items (shows/produces/apps) that users are likely to interact with.

These learned models are typically trained, and evaluated, based on the items that users have interacted with in the past. However the users’ previous interaction behaviour can be affected by what items are being promoted to users by any already deployed recommender (aka item-selection bias). For instance, users may only interact with those items that they have been exposed to by the system. In this way, using historical interactions in the training and evaluation of a new recommender model, where those interactions have been obtained from the deployed recommender system, forms a closed feedback loop, i.e. the deployed recommender system has a direct effect on the collected feedback. The systems often also make use of an internal feedback loop in order to refine information needs or diversity results.

We pursued three primary directions in this work package, which are detailed below. We also enhanced our open-source tooling to facilitate this research.

Closed-loop Feedback in Recommendation and Search Data

Recommendation and search systems are often trained and evaluated based on feedback data that were collected from a deployed engine. This causes a feedback loop in which the recommender/search system and user are strongly connected in a dynamical system; each influences the other and their dynamics. Meanwhile, the output of this system is collected and used for training and evaluation of new systems.

We quantified the effects of such closed loops; we found that they exacerbate the effects of popularity bias [1] and lead to Simpson’s Paradox during evaluation [5]. We then provided practical approaches for overcoming these problems through randomly opening the feedback loop [3] and propensity-based stratification during evaluation [5]. We also provided a new theoretical link between explicit and implicit feedback in recommender systems [2]. Further, we explored the presence of feedback loops in search datasets. Here, the set of queries is affected by survivorship bias, since those that the search system is unable to answer are discarded [14]. We quantified the effects of this closed loop on the training and evaluation of machine learning models and provided concrete suggestions on how to lessen the negative effects of such feedback loops when building datasets in the future [13,14].

Closed-loop Feedback in Proactive Search

Query strings entered into search engines often under-specify a user’s information need; query terms that could help identify the most relevant information may not be included for ease of writing, to fit into natural language, or simply because the terms themselves are what the user is looking to find. To overcome this, search engines often employ a feedback loop, wherein key terms from the top-scoring results are proactively fed back into the query in a process called pseudo relevance feedback. After one or more iteration of this process, the final results are presented to the user. This process creates closed-loop feedback because the retriever is affected by the results of the scorer. 

Previously, pseudo relevance feedback was limited to heuristic-based lexical matching systems. Through CLDS, we explored the application of learned retriever and scoring functions. Specifically, we found that the feedback mechanism can be applied to the latest dense retrieval methods [11,19], enhancing the precision and recall of search results. It is also effective when applied using external data for feedback in a zero-shot setting [20]. Further, we found that the feedback loop itself can be modelled as an information-seeking agent, which needs to decide whether to explore results from an initial candidate set or exploit the nearby documents of the top-scored results [15,16]. Finally, we explored approaches for automatically reformulating ambiguous queries, allowing the results to cover multiple potential information needs a user may have [6]. In all these formulations of result feedback loops, we find that the feedback loop enhances the results, identifying more relevant documents for user queries.

Closed-loop Feedback in Conversational Recommendation:

Conversational recommender systems (CRSs) have recently received much attention for addressing the information asymmetry problem in information seeking, owing to their flexible recommendation strategies and their natural multi-turn decision-making processes. In particular, a conversational recommender system is a type of recommender systems that can elicit the dynamic preferences of users and take actions based on their current needs through real-time multi-turn interactions. To this end, the conversational recommender systems can be generally considered to form closed loop systems in which the inputs (i.e. users' feedback) of the recommender systems are fully or partially determined by the outputs (i.e. recommended items).

WP2: Closed loop mass spectrometry measurement and analysis of metabolomics

Metabolomics is the study of metabolites, small molecules that occur in all living organisms. Collecting and understanding metabolomics data is challenging. Data can be collected by putting a sample through a liquid chromatography Mass Spectrometry (LC-MS) system, in which the molecules present are first separated by the chromatography before being analysed by the mass spectrometer. The result is a series of (typically 1000s) of Mass Spectrometry scans, each of which records the mass to charge ratio and intensities of the metabolites present at a particular chromatographic retention time. Unfortunately, it is almost always impossible to identify molecule from their mass alone and it is therefore common to fragment metabolites via interleaving the normal MS scans with targeted MS/MS scans. Each MS/MS scan can provide information that can aid in the structural elucidation of one or more molecules. A key challenge when using mass spectrometry to collect and understand metabolomics data is choosing where and when to prioritise the MS/MS scans, while still maintaining enough normal MS scans be able to carry out basic data analysis. Acquiring improved data will benefit experiments, but it could be at the cost of missing something that would have otherwise been collected, potentially damaging experiments and trials.

Closed loop issues become a problem when we attempt to make real-time intelligent decisions, rather than using pre-defined scan scheduling rules. When trying to make real-time closed loop decisions, two loops affect how we prioritise scans, as can be seen in the figure. Firstly, there is a short-term within sample loop (blue arrow), where we feedback the information gained from each scan in order to determine the best next scan type and location. Secondly, there is a loop where the results from previous sample are fed back into the decision process (red arrow), providing information which can help guide scan choices in future samples. Within this general measurement framework there are several inherent closed loop issues. The first is the finite scan budget: we have a limited number of scans and they cannot be done simultaneously. This must be done with only partially observed time series and with no knowledge of the success of our choices until after the completion of the sample (lagged reward). Within the second, outer loop we see issues with exploration vs exploitation, where we have the choice between confirming metabolites we think we know (exploitation) or attempting to fragment new, interesting metabolites (exploration). Balancing this trade-off well could lead to improved studies, while doing it badly could result in biased results which could invalidate entire studies.

Work has involved designing a framework which allows us to design controllers which can be used to control scans in real-time in both the Mass Spectrometer and in simulation [1]. Using this framework, new and improved controllers have been designed which have been shown to improve the number of metabolites fragmented [2] (inner loop). Current work is focused on how we can apply these methods across multiple samples and correct timing differences across samples in real-time to improve performance (outer loop).

Publications

[1] Wandy, J., Davies, V., JJ van der Hooft, J., Weidt, S., Daly, R., & Rogers, S. (2019). In silico optimization of mass spectrometry fragmentation strategies in metabolomics. Metabolites, 9(10), 219.

[2] Vinny Davies, Joe Wandy, Stefan Weidt, Justin JJ Van Der Hooft, Alice Miller, Rónán Daly, and Simon Rogers. “Rapid development of improved data-dependent acquisition strategies”. In: Analytical chemistry 93.14 (2021), pp. 5676–5683.

Closed-loop optimisation of hearing aid parameters, based on subjective user feedback

WP2: Closed-loop hearing aid optimisation

The Problem

466 million people worldwide have disabling hearing loss due to genetic causes, complications at birth, certain infectious diseases, chronic ear infections, the use of particular drugs, exposure to excessive noise, and ageing (source: WHO). A hearing loss impacts an individual’s ability to communicate (effectively) with others, leading e.g. to academic and adjustment problems for children. Exclusion from communication can have a significant impact on everyday life, causing feelings of loneliness, isolation, frustration and dependence. Unaddressed hearing loss poses an annual estimated global cost of $750 billion (12th most common contributor), including health sector costs and productivity.

Modern hearing aids can partially compensate for a hearing loss. However, the effectiveness of a hearing aid is highly dependent on a suitable configuration of the advanced medical device. A state-of-the-art hearing aid requires 10s of different algorithms and 1000s of parameters to compensate for even the simplest hearing loss; each parameter configuration must be tuned to the specific user and condition. The complexity of hearing loss and the HA itself have traditionally called for experts to tune the device in clinical conditions based on audiograms and informal verbal communication with the patient with little control left to the user when leaving the clinic.

Research Aim:

We will enable real-time and context-dependent optimisation of hearing aid configurations to empower the HA-user and ultimately improve the user/listening experience. By exploiting the availability of population-wide, real-time data streams we will investigate real-time collaborative modelling, intervention and optimisation strategies to provide a significant quality improvement and speed-up in the closed-loop optimisation for both individual and the population of HA-users as a whole.

RQ1 – User Modelling and Analysis:

Models for hearing/hearing loss and choice of configuration are typically formulated and derived based on a single - or a few - patients. By continuously collecting millions of observations from users and clinicians about HA-use and configurations, we will develop scalable non-parametric hierarchical Bayesian models of users, contexts and configurations which enables group-based analysis of preference structures. It will provide predictive models for HA-configurations (e.g. based on user demographic, audiogram and behaviour) and the probabilistic nature of the models will support robust and informed clinical decision-making. Based on the models we will extract and analyse user and preference patterns related to current HA-use, configurations and contexts.

RQ2: Closed-loop collaborative preference elicitation and optimisation:

It is possible to elicit, model and optimise subjective preference in the HA-domain using Bayesian modelling and reinforcement learning (/Bayesian optimisation) in a closed-loop fashion. However, this has been done for individuals independently of other peoples preferences, configurations and context. We will develop new techniques to support multiple feedback loops, multiple timescales and asynchronous feedback to support the collaborative setting with input from multiple users and clinicians at differences timescales and with mixed fidelity. We will develop multi-fidelity/confidence-based optimisation and intervention strategies which accounts for user consistency and reliability in order to avoid suboptimal learning and biases in the closed-loop and collaborative setup. We will develop Bayesian optimisation methods supporting multi-objective and engaging optimisation taking into the intention of the user and costs involved in asking users questions or consulting a clinician. We expect to evaluate the models and strategies in several empirical studies to investigate the effect and consequence of the collaborative and closed-loop optimisation.

Publications:

[1] Salman Mohammadi, Anders Kirk Uhrenholt and Bjørn Sand Jensen, Odd-One-Out Representation Learning, NeurIPS 2020: Workshop on Object Representations for Learning and Reasoning.
[2] Lisa Laux, Marie F.A. Cutiongco, Nikolaj Gadegaard and Bjørn Sand Jensen. Interactive machine learning for fast and robust cell profiling, PLoS ONE, 2020.
Anders Kirk Uhrenholt and Bjørn Sand Jensen, Efficient Bayesian Optimization for Target Vector Estimation, AISTATS 2019.
[3] Anders Kirk Uhrenholt, Valentin Charvet and Bjørn Sand Jensen, Probabilistic Selection of induction point in sparse Gaussian process models, Proceedings of the Thirty-Seventh Conference on Uncertainty in Artificial Intelligence. Uncertainty in Artificial Intelligence. ISSN: 2640-3498. PMLR, Dec. 1, 2021, pp. 1035–1044.
[4] Jasper Kirton-Wingate, Modelling Contextual Preference for Hearing Aid Settings - A Machine Learning Approach, Master of Science by Research, University of Glasgow, 2020, https://theses.gla.ac.uk/81380/
[5] Krasimir Ivanov, Anders Kirk Uhrenholt, Sebastian Stein, Bjørn Sand Jensen. Accelerated GP-based ABC with improved surrogate modelling. Under review at UAI2023. 2023.
[6] Lisa Laux, Marie F. A. Cutiongco, Nikolaj Gadegaard, and Bjørn Sand Jensen. “Interactive machine learning for fast and robust cell profiling”. In: PLOS ONE 15.9 (2020), e0237972.
[7] Richard Sonntag. “A Multi-Task Approach to Hearing Aid Fine-Tuning”. MSci. University of Glasgow, 2021.
[8] Anders Kirk Uhrenholt. “Assumptions and efficiency in Gaussian process modelling”. PhD Thesis. University of Glasgow, 2021. url: https://theses.gla.ac.uk/82442/.
[9] Anders Kirk Uhrenholt and Bjørn Sand Jensen. “Efficient Bayesian Optimization for Target Vector Estimation”. In: Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics. The 22nd International Conference on Artificial Intelligence and Statistics. ISSN: 2640-3498. PMLR, Apr. 11, 2019, pp. 2661–2670.

WP3: Intermittent Control in Data Science

Alberto Álvarez, Henrik Gollee, Roderick Murray-Smith

Variability is an important concept in control systems and data science. It is commonly associated with exploration of the state-space, finding new solutions, and with adaptation and learning. A common way to represent variability is to use stochastic models and added noise, but this has its pitfalls in closed loop data systems as added noise can be propagated around a continuous feedback loop, masking causalities and making identifying system properties more difficult or impossible.

Event-driven control systems naturally introduce variability, e.g. by breaking the continuous feedback loop. This can lead to periods of open-loop exploration of the state-space, making it easier to identify causalities. Intermittent Control (IC), which switches between open and closed-loop regimes, introduces variability through event-driven control while allowing the identification of causal relationships between data sources, and provides clearly defined trigger points for process interactions to take place.

Our IC framework is based on a generalisation of the Model Predictive Control (MPC) approach, using a limited prediction horizon and generalised models for the predictive hold element. In combination with event triggering, this provides a version of MPC which is more suitable for applications in closed loop data science problems.

While the IC framework is based on mechanistic state-space control approaches, it can be extended to provide capabilities for data driven identification and control. This includes machine learning approached for the internal model and for triggering (e.g. Gaussian processes, neural networks and Monte Carlo simulations), causal identification techniques, and data driven adjustment of control gains (e.g. using reinforcement learning).

Main outcomes:

• We have evaluated intermittent control in the context of closed loop system identification and demonstrated that by only using feedback information intermittently, the propagation of noise around the loop is inhibited. In the context of data science, this addresses the problem that an interaction with the system could be propagated through the feedback loop, affecting current and future actions. IC can break this propagating loop through event triggers, allowing adjustment between feedback interaction and propagation of information around the loop.
• Intermittent control has been evaluated in the context of classical Human-Computer Interaction (HCI) tasks. In addition to the overall underlying human control behaviour, intermittent control can also model aspects of human variability in the context of a one-dimensional pointing task [1].
• The Intermittent Control framework has been extended to be fully data-driven, rather than requiring knowledge of the system to be controlled:
• The “Internal model” (i.e. the intermittent hold) and the predictor have been implemented as Gaussian process models (GPs). Data driven identification of these components have been evaluated in various dynamic systems, including a swing-up inverted pendulum model. [2]
• Data-driven methods to identify the “Control gains” have been implemented and evaluated, including reinforcement learning and controller designs based on system identification.
• We have implemented and evaluated an adaptive intermittent control framework which exploits intermittent triggering to address the stability – plasticity dilemma (exploitation vs exploration) in adaptive closed loop systems [3].

The initial implementation of intermittent control was developed in MATLAB (The Mathworks, USA) which makes it accessible to an engineering audience. To allow data scientists access to these algorithms the IC implementation was transferred to Python. This also allows easy integration with the rich eco-system of machine learning tools of the Python environmen. The Python code for the implementation of our intermittent control model is available at https://.....

Publications

[1] Álvarez Martín, J.A., Gollee, H., Müller, J., Murray-Smith, R., 2021. Intermittent Control as a Model of Mouse Movements. ACM Trans. Comput.-Hum. Interact. (TOCHI) 28, 35:1-35:46. https://doi.org/10/gmpchf

[2] Álvarez Martín, J.A., Doublein, T., Gollee, H., Murray-Smith, R., 2022. Understanding the variability of pointing tasks with event-driven intermittent control, in: Proc. MTNS 2022. Presented at the 25th International Symposium on Mathematical Theory of Networks and Systems, Bayreuth, Germany.

[3] Álvarez-Martín, J.A., Gollee, H., Gawthrop, P.J., 2023. Event-driven adaptive intermittent control applied to a rotational pendulum. Proceedings of the Institution of Mechanical Engineers, Part I: Journal of Systems and Control Engineering. https://doi.org/10.1177/09596518221147340

WP3: Closed loop issues in Urban traffic control

Traffic congestion has been a key urban issue, causing significant costs on economics in many cities worldwide. The traffic signal control system, as an essential tool of smart road principles, can be used to reduce the level of congestion, and thereby transport emissions and fuel consumption in an urban area. The adaptive traffic signal control approach like SCOOT (Hunt et al., 1982) and SCATS (Sims and Dobinson, 1980) has been widely used in real-world traffic network. They are based on an open loop control system that does not consider feedback control in the traffic network. In contrast, a learning-based method like a reinforcement learning (RL) algorithm can be employed to learn from the traffic environment by taking actions (e.g., switch signal phasing) and observing the feedbacks (e.g., queue lengths), enabling researchers and planners to predict the traffic flow more accurately, thereby optimize the traffic signal plan.Due to the new technology improvements, high-tech vehicles (e.g., Connected and Automated Vehicle (CAV)) can communicate with traffic infrastructure, generating important new data. For example, planners can optimize traffic signal control plans based on more accurate current traffic flows and future traffic flow predictions by using data (e.g., speed, location, headway, etc.) sent from CAVs to traffic signal controls. In addition, CAVs can drive more efficiently by receiving updated traffic signal plans, potentially improving an overall network system performance (e.g., number of stops, fuel consumptions, etc). Unfortunately, research on the adoptive traffic signal control with a RL approach and the applications of CAVs in this environment is scarce. In this WP, we use the road intersection as a RL agent while allowing the CAVs to adjust their speed adaptively based on the real-time signal plan received from the controller to examine the closed-loop effects of learning traffic signal controllers and CAVs on the network system performance. Preliminary results show that the proposed method outperforms other dynamic traffic signal control strategies in terms of average vehicle delay and queue length. The sensitivity analysis with different market penetration rates of CAVs and traffic saturation degrees is in the process.

Publications