The research area “Medical Image Processing, Modeling and Simulation based on Artificial Intelligence” (MIMAS.ai) covers highly dynamic topics. These topics are gaining importance in medical applications, mainly due to recent technological advances.
Medical image data are used for diagnosis, treatment planning, intervention monitoring, and documentation. Common modalities include 2D images such as X-rays, 3D scans such as CT or MRI, and even video sequences capturing temporal changes. To process these multimodal data, the Medical Informatics Research Unit develops methods for image analysis and segmentation. Using AI-based methods, the data are registered and segmented to extract patient-specific anatomical structures such as vessels, tissue, or skin. These segmented structures then form the basis for building medical models and simulations.
However, the success of machine learning methods depends heavily on the quality and quantity of training data. Especially in medicine, suitable data are often missing or restricted due to privacy concerns. Therefore, research also focuses on generating ground truth data more efficiently. For example, CycleGANs are used to create synthetic training data, One Shot Learning expands datasets through augmentation, Transfer Learning applies existing models to similar problems, and Domain Adaptation adjusts models to new data distributions.
Deep Learning for the extraction of anatomic structures from patient cohorts, e.g., for abdominal aortic aneurysms.
Learned Visual Representations of the BRATS dataset from Diffusion (left) and derived segmentations for only ten training slices (right).
Deep Learning based segmentation of mandible, facial bone, teeth, mandibular canal and metal structures, to support patient specific implant design.
Modeling provides simplified representations of reality. In medicine, these models must be clinically relevant and based on available data. They range from 3D surface models to complex blood flow models. For example, virtual 3D patient avatars support diagnostics in burn medicine, chronic wound management, or forensics. Physicians can measure wound size, monitor healing, and objectively document treatment progress.
Biomechanical simulations reproduce processes such as blood flow. Based on registered and segmented data, anatomical models are created, meshes are generated, and material properties are defined. From these models, quantitative features like vessel diameters or wall stresses are derived. Consequently, simulations help experts make better decisions. For instance, they can estimate the rupture risk of aneurysms or assess the effectiveness of stents. Moreover, simulators based on these models provide valuable training opportunities for physicians.
Deformed beam as an example of an ML-based surrogate model (Perceiver-IO) for solving elasticity equations. The dots correspond to the surrogate predictions, while the solid lines originate from the exact numerical simulation (FEM).
Transfer of a burn wound from a 2D image onto a 3D patient model. The automatic wound localization and adaptation of the 3D model to the patient’s body shape enables a more objective estimation of the wound size and documentation of the healing process.
Trust in machine learning applications is essential in medicine. Physicians and patients must rely on both the database and the prediction models. Therefore, MIMAS.ai emphasizes methods for validating data, interpreting predictions, and analyzing deviations.
The Medical Informatics Research Unit develops complete data processing frameworks to support physicians in clinical practice. Use cases include:
In addition to structured data, the research also incorporates images, videos, and signals. A key focus lies on explainable AI methods that ensure interpretability and transparency — both crucial for acceptance in medical environments. For example, CaTabRa, a Python package, allows automated analysis of tabular data. It supports descriptive statistics, out-of-distribution detection, prediction model training, and model evaluation.
CaTabRa is a Python package for analyzing tabular data in a largely automated way. This includes generating descriptive statistics, creating out-of-distribution detectors, training prediction models for classification and regression tasks, and evaluating/explaining/applying these models on unseen data.
The research fields of the Medical Informatics Research Unit are very closely related. Medical image data often form the basis for modeling, models in turn form the basis for medical image processing and information extraction, as well as for the simulation of processes in the human body. The basic technologies and methods used in the research fields also show a variety of overlaps. GPU-based (Graphics Processing Unit) parallel computations have enabled the triumph of Deep Learning in image processing in recent years, and at the same time provide the basis for the simulation of processes in the human body.
However, physiological interactions require corresponding models of anatomical structures, which are extracted from medical image data using segmentation methods. Registration – the computation of a transformation that brings multiple data sets (model, image, volume) into geometric agreement – enables the use of multiple data sources as well as the transfer of information between different data domains. Information extraction is performed in different ways in all research fields. The following medical application examples illustrate the interrelationship of these research fields:
Aneurysms are typically diagnosed using CTA (computed tomography angiography) scans. Segmentation methods identify the aneurysm and surrounding blood vessels. From this data, a volume model (mesh) for blood flow simulation is generated. The simulation enables the calculation of blood pressure and vascular wall stress. Furthermore, by analyzing a cohort of patients — for example, aneurysm cases from the last ten years — machine learning methods can detect rupture risks. These insights support physicians in selecting the most appropriate treatment strategy for each patient.
Patients with burn wounds usually receive initial treatment in the emergency room. Using medical modeling techniques, a virtual 3D body surface model is created and adapted to the patient via an RGB-D scan. Image analysis methods classify the burn depth, while the extent and severity of the wounds are documented on the surface model. In addition, the temporal progression of wound healing is continuously recorded. This information allows not only more precise monitoring of the individual case but also contributes to improving the treatment of future patients.
The overarching aim of all these efforts is the broader adoption of individualized and evidence-based medicine. To achieve this, current research methods must be further developed at an early stage in close cooperation with medical experts. Only then can innovative techniques successfully transition into clinical practice in the medium term and deliver real benefits for patients.
This project is financed by research subsidies granted by the government of Upper Austria. RISC Software GmbH is Member of UAR (Upper Austrian Research) Innovation Network.
Head of Research Unit Medical Informatics
