The research area “Medical Image Processing, Modeling and Simulation based on Artificial Intelligence” (MIMAS.ai) covers a cross-section of highly dynamic research topics, which are becoming increasingly important in medical application fields, not least due to current technological advances.
Medical image data are used in a variety of forms for diagnosis, treatment planning, monitoring of interventions and observation of changes in condition and documentation. Common image modalities range from 2D images (X-ray, wound photos…) to 3D scans (computed tomography, magnetic resonance imaging, digital subtraction angiography…) to videos (2D+t, 3D+t), which also capture the temporal course. In order to process these multimodal image data for medical diagnosis, treatment and documentation purposes, the Medical Informatics Research Unit is developing various methods for image analysis and image segmentation. Using artificial intelligence-based methods, these image data are registered (overlaid) in order to subsequently segment (extract) patient-specific anatomical structures, such as blood vessels, tissue or skin. These segmented anatomical structures form the basis for medical models and simulations based on them.
The success of machine learning methods for registration and segmentation depends largely on the number and quality of available training data. Especially in medical application fields, data with corresponding ground truth are often missing or they are not available due to data protection reasons. Therefore, research in this area also focuses on methods that allow an easy and interactive generation of a ground truth. These methods include CycleGANs to generate synthetic training data, One Shot Learning to generate a multitude of training data from one data set by means of augmentation, Transfer Learning, whereby already learned knowledge models are used for similar problems, AI-based image processing, modeling and simulation of medical data Medical Image Processing, Modeling and Simulation based on Artificial Intelligence77 or Domain Adaptation to adapt learned models to new data distributions.
Deep Learning for the extraction of anatomic structures from patient cohorts, e.g., for abdominal aortic aneurysms.
Learned Visual Representations of the BRATS dataset from Diffusion (left) and derived segmentations for only ten training slices (right).
Deep Learning based segmentation of mandible, facial bone, teeth, mandibular canal and metal structures, to support patient specific implant design.
Modeling is generally understood to be a simplified representation of reality. In the medical context, models are strongly oriented towards clinical relevance, applicability and available data. Models range from 3D body surface models to blood flow models. In many medical fields, such as burn medicine, chronic wound management or forensics, the affected body surface of patients is an essential diagnostic basis. In the research focus, virtual 3D patient avatars are created, which match real patients as closely as possible and thus enable objective diagnostics and medical (progress) documentation. For example, the size and position of chronic wounds can be precisely determined and the success of the therapeutic approach can be assessed over time.
Biomechanical simulation can be used to reproduce processes in the human organism (e.g., blood flow). Based on registered and segmented medical image data, methods for modeling anatomical structures (blood vessels, tissue, skin…) are used, computational grids (meshes) are created and material properties (elasticity, viscosity…) are determined. Based on the model, geometric features (e.g., maximum diameter of a blood vessel) as well as simulation-based features (e.g., mechanical stress of a vessel wall) can be calculated.
These simulations allow medical experts to make or evaluate diagnoses and treatment decisions using quantitative measures. For example, blood flow simulation can be used to determine the risk of rupture of an aneurysm and to evaluate the effectiveness of interventions in the vascular system (e.g., clipping of aneurysms, insertion of a stent-graft). These models and simulations are used, for example, in simulators for the training of physicians and for the realistic training of medical interventions.
Deformed beam as an example of an ML-based surrogate model (Perceiver-IO) for solving elasticity equations. The dots correspond to the surrogate predictions, while the solid lines originate from the exact numerical simulation (FEM).
Transfer of a burn wound from a 2D image onto a 3D patient model. The automatic wound localization and adaptation of the 3D model to the patient’s body shape enables a more objective estimation of the wound size and documentation of the healing process.
A key success factor for the application of machine learning in medicine is the trust of physicians and patients in the database and the prediction models derived from it. This trust is based on methods for validating the data, interpreting the model predictions and analyzing deviations. By making these methods available and usable in a data processing and analysis framework that encompasses the entire data processing pipeline, the Medical Informatics Research Unit supports physicians in clinical practice.
The development of the necessary data processing structures takes place within the framework of selected example scenarios. Application areas include transfer management in the intensive care unit, optimization of the Manchester triage system in the emergency room, optimal application of blood reserves, and prediction of cardiac instability.
The focus is on researching a generic and easily configurable data processing chain for proven and latest methods in order to meet future requirements. In addition to structured data, image, video and signal data as well as combinations of different data modalities and specialized feature extraction serve as the information and data basis. The research focus is on interactive data analysis of different modalities with a focus on interpretability and traceability of single data or data groups in terms of “Explainable AI.” Especially in medicine, transparency is of particularly high importance for the acceptance of AI-based software.
CaTabRa is a Python package for analyzing tabular data in a largely automated way. This includes generating descriptive statistics, creating out-of-distribution detectors, training prediction models for classification and regression tasks, and evaluating/explaining/applying these models on unseen data.
The research fields of the Medical Informatics Research Unit are very closely related. Medical image data often form the basis for modeling, models in turn form the basis for medical image processing and information extraction, as well as for the simulation of processes in the human body. The basic technologies and methods used in the research fields also show a variety of overlaps. GPU-based (Graphics Processing Unit) parallel computations have enabled the triumph of Deep Learning in image processing in recent years, and at the same time provide the basis for the simulation of processes in the human body. However, physiological interactions require corresponding models of anatomical structures, which are extracted from medical image data using segmentation methods. Registration – the computation of a transformation that brings multiple data sets (model, image, volume) into geometric agreement – enables the use of multiple data sources as well as the transfer of information between different data domains. Information extraction is performed in different ways in all research fields. The following medical application examples illustrate the interrelationship of these research fields:
Aneurysms are diagnosed in CTA (computed tomography angiography) scans. Segmentation of aneurysms and blood vessels is used to create a volume model (mesh) for blood flow simulation. The simulation allows calculation of pressure and vascular stress. For a cohort of patients (e.g., aneurysm patients over the last ten years), these characteristics can be used to determine the risk of rupture using machine learning-based data analysis and to select an appropriate treatment strategy based on this.
Patients with burn wounds are initially treated in the emergency room. Using medical modeling methods, a virtual 3D body surface model is adapted to the patient using an RGB-D scan. Burn depth is classified using image analysis methods. The wounds (extent and depth) are documented on the body surface model. The temporal progression of the wound due to subsequent treatment is documented on the body surface model and can thus be used to improve the treatment of future patients.
The common goal of all work is the further spread of individualized and evidence-based medicine. To this end, current research methods must be further developed at an early stage in cooperation with medical experts. This is the only way to ensure that current methods are also used in clinical practice in the medium term for the benefit of patients.
This project is financed by research subsidies granted by the government of Upper Austria. RISC Software GmbH is Member of UAR (Upper Austrian Research) Innovation Network.
Head of Research Unit Medical Informatics
