Skip to Content
← Back to services

Software framework for the comparison and benchmarking of AI and traditional algorithm

TEF-Health Service
Virtual
Unique Identifier: 7a96d86d-07a2-48f9-bfa1-35affbe3b6f0

Service Description

Overview

This service provides an open-source software framework designed to standardize and simplify the evaluation, comparison, and integration of machine learning models alongside traditional algorithms. When working with complex or multi-modal biomedical sensor data, standard evaluation tools often struggle to handle non-tabular data structures and custom processing steps. By utilizing an object-oriented paradigm, the framework enables teams to build modular dataset interfaces and structured execution pipelines. The primary objective is to streamline validation workflows, prevent implementation errors during nested cross-validation or parameter optimization, and provide a unified benchmark platform for comparing legacy algorithms with modern data-driven approaches.

How can the service help you?

Evaluating non-standard machine learning pipelines and comparing them with conventional heuristics often leads to complex, custom codebases prone to data leakage and validation errors. This service establishes an object-oriented architecture to structure your datasets and algorithm steps cleanly.

  • Before: Fragile, unstandardized script setups where comparing machine learning workflows against traditional algorithmic baselines requires manual, error-prone evaluation logic.
  • After: A robust, modular framework with unified interfaces for parameter optimization, cross-validation, and standardized performance comparison.

It allows development teams to accelerate algorithm benchmarking, enforce reproducible evaluation standards, and maintain clean abstractions across heterogeneous code bases.

How will the service be delivered?

The service is delivered through software access, technical documentation, and collaborative setup support. Clients provide details regarding their multi-modal data structure and algorithm requirements. Our team assists in configuring the software architecture, defining custom dataset classes, and setting up standardized validation pipelines.

Additional information

Provider description

Operating from the Department of Artificial Intelligence in Biomedical Engineering at Friedrich-Alexander-Universität Erlangen-Nürnberg, we are a service provider node within the TEF-Health consortium. The research group specializes in machine learning, biomedical signal processing, and multimodal sensor synchronization. Our team provides testing infrastructure and scientific support for evaluating medical devices, wearables, and contactless sensing systems.

Technical details

The service relies on tpcp (Tiny Pipelines for Complex Problems), an open-source Python framework tailored for complex algorithm evaluation.

  • Input Data Requirements: Complex, multi-modal, or time-series datasets requiring custom data structures and custom splitting/indexing logic.
  • Core Tasks & Processing: Implementation of object-oriented dataset interfaces, creation of modular algorithm pipelines, parameter optimization, nested cross-validation, and performance benchmarking across diverse model types.
  • Outputs: A structured software framework containing object-oriented data loaders, standard pipeline interfaces, and reproducible evaluation tools tailored to client datasets.
  • Reference & Code Base: https://github.com/mad-lab-fau/tpcp

Service customization

The engagement can be tailored to the client's current software stack. SMEs and startups can choose to adopt individual components (e.g., utilizing specific dataset helpers) or engage in building full parameter optimization and multi-model benchmark suites customized for their proprietary algorithms.

Offerings: Platform (trusted research environment, authentication federation, etc.)
Provider Logo

Provider & Contact

Provider Organisation Friedrich-Alexander-Universitaet Erlangen-Nuernberg (FAU)
Provider Country Germany
Ask about this service
Billing: per hour
Full Price 120 EUR
Reduced Price No discount can be provided
Pricing Detail

The software is free, the price indicated is only for the consulting/support. This price by hour is an estimate.

Operational Details

Service Inputs Complex/Multimodal Dataset(s)
Service Outputs Framework for standardized data handling, ML Pipelines, and algorithm evaluation
Certification Support None

Related services

  • 3T MRI Human imaging data acquisition (clinical)

    Centre Hospitalier Universitaire De Rennes (CHU RENNES)

    MRI 3 Tesla (Philips Ingenia Edition X), imaging, data acquisition, clinical

    View service →
  • 3T MRI Imaging data acquisition (Clinical)

    Karolinska Institutet (KI)

    Centre for Imaging Research (CIR) is a world-leading imaging centre at Karolinska Institutet in Sweden in collaboration with SciLIfeLab and RISE. It offers an exceptional and unique collection of core facilities for cutting-edge structural, functional, and metabolic in vivo multimodal imaging and endovascular/minimally invasive surgical techniques. The core facility, MRI Centre, aims to catalyse novel and translational research using MRI. We offer a state-of-the-art 3 Tesla GE SIGNA Premier system-wide bore gantry and high-performance XT gradients. We support all types of MRI studies in humans, ranging from small single-centre projects to large-scale international multi-centre studies. A special focus is on offering MRI suitable for multi-modal integration such as CT, PET, EEG and MEG. The KI MRI Centre also offers a full-service lab for drawing blood and investigating psychophysiology (EDA, PPG, Eye-tracking) along with standardised image post-processing and data storage facilities with cluster computing for data analysis. In this service, we offer full support in project consultation, ethical permits and module technical support. For more information, visit the website: https://imagingresearch.se/facilities/corefacilities/mrc

    View service →
  • 3T MRI Imaging data acquisition (Pre-clinical, Large animal)

    Karolinska Institutet (KI)

    Centre for Imaging Research (CIR) is a world-leading imaging centre at Karolinska Institutet in Sweden in collaboration with SciLIfeLab and RISE. It offers an exceptional and unique collection of core facilities for cutting-edge structural, functional, and metabolic in vivo multimodal imaging and endovascular/minimally invasive surgical techniques. The core facilities, Karolinska Experimental Research and Imaging Centre (KERIC) and the MRI Centre, aim to catalyse novel and translatory research using MRI. At KERIC, we offer a 9.4T MRI for animals such as rabbits and small piglets (see service for small animal 9.4T MRI). A state-of-the-art 3 Tesla GE SIGNA Premier system-wide bore gantry and high-performance XT gradients are available for large animal preclinical studies. We offer specialised services for large animal models as well as ex vivo imaging and support all types of MRI studies in humans, ranging from small single-centre projects to large-scale international multi-centre studies. A special focus is on offering MRI suitable for multi-modal integration such as CT, PET, EEG and MEG. The KI MRI Centre also offers a full-service lab for drawing blood and investigating psychophysiology (EDA, PPG, Eye-tracking) along with standardised image post-processing and data storage facilities with cluster computing for data analysis. In this service, we offer full support in project consultation, ethical permits, module technical support and animal technicians. For more information, visit the website: https://imagingresearch.se/facilities/corefacilities/mrc Or https://imagingresearch.se/facilities/corefacilities/keric

    View service →
  • 7T MRI Imaging data acquisition (Clinical)

    Karolinska Institutet (KI)

    Centre for Imaging Research (CIR) is a world-leading imaging centre at Karolinska Institutet in Sweden in collaboration with SciLIfeLab and RISE. It offers an exceptional and unique collection of core facilities for cutting-edge structural, functional, and metabolic in vivo multimodal imaging and endovascular/minimally invasive surgical techniques. The core facility, MRI Centre, aims to catalyse novel and translational research using MRI. A unique next-generation clinical 7 Tesla Siemens MAGNETOM Terra.X. equipped with an ultra-high field scanner also offers cutting-edge gradient performance (130 mT/m at 250 T/m/s), parallel transmission, as well as deep learning-based acceleration and denoising for improved image quality. Micrometre resolution in vivo for microstructural imaging and layered fMRI. We support all types of MRI studies in humans, ranging from small single-centre projects to large-scale international multi-centre studies. A special focus is on offering MRI suitable for multi-modal integration such as CT, PET, EEG and MEG. The KI MRI Centre also offers a full-service lab for drawing blood and investigating psychophysiology (EDA, PPG, Eye-tracking) along with standardised image post-processing and data storage facilities with cluster computing for data analysis. In this service, we offer full support in project consultation, ethical permits and module technical support. For more information, visit the website: https://imagingresearch.se/facilities/corefacilities/mrc

    View service →
  • 9.4T MRI Imaging data acquisition (Pre-clinical, Small animal )

    Karolinska Institutet (KI)

    Centre for Imaging Research (CIR) is a world-leading imaging centre at Karolinska Institutet in Sweden in collaboration with SciLIfeLab and RISE. It offers an exceptional and unique collection of core facilities for cutting-edge structural, functional, and metabolic in vivo multimodal imaging and endovascular/minimally invasive surgical techniques. The core facility Karolinska Experimental Research and Imaging Centre (KERIC), we have a preclinical 9.4T Bruker USR 94/30 MRI system. The state-of-the-art 9.4T MRI is equipped with single-mouse, single-rat, or double-mouse beds. It has three gradient bore sizes to arrange an effective inner diameter of 20cm (large animals), 12cm (medium animals) or 6cm (small animals). Methods in anatomical imaging, fMRI, Ex vivo, Pharmacological MRI and Cardiac imaging (CINE). For the mouse brain, a 4-channel phased array cry-coil drastically increased signal-to-noise. In this service, we offer full support in project consultation, ethical permits, module technical support, animal technicians and data analysis. We can host animals directly or indirectly through collaboration within KU (AKM) and other animal labs. For more information, visit the website: https://imagingresearch.se/facilities/corefacilities/keric

    View service →
  • Access to Compound Collection

    Karolinska Institutet (KI)

    The Unit: Chemical Biology Consortium Sweden (https://www.scilifelab.se/units/cbcs/) The infrastructure is part of Science for Life Laboratory (SciLifeLab) which is an academic collaboration between Swedish universities (including Karolinska Institutet) and a national research infrastructure with a focus on life science. Chemical Biology Consortium Sweden (CBCS) is a national research infrastructure offering an extensive portfolio of services - from computational and organic chemistry to phenotypic small molecule screens in cells and model organisms, cell painting and functional precision medicine. The mission of CBCS is to provide a state-of-the-art platform and expertise for the generation of high-quality bioactive chemical tools (small organic molecules) for applications within life science research in general and with the ultimate goal to explore complex biology. Our units comprise compound handling labs (hosting the SciLifeLab Compound Collection of ca 350,000 molecules), screening labs (microtiter format liquid handling robotics and a wide range of plate readers), and chemistry labs (equipment and instrumentation for organic chemistry synthesis and analysis). The service: Access to compound collection https://www.cbcs.se/the-compound-collection The Scilifelab Compound Collection is managed by the Compound Center at CBCS KI, and is based on donations of high-quality sets from the pharmaceutical industry (Biovitrum, Karobio, Orexo/Biolipox, KDev) and has been developed and expanded with compounds from various commercial compound vendors. The compounds can be readily available in assay-ready plates through acoustic nano-litre scale liquid dispensing or in 96-tube racks; compounds are stored at 10mM concentration in DMSO solution at - 20 °C under humidity control. We provide: o Management, storage, and processing of solids and liquids. o Compound delivery and interface with key customers. o Performance of instruments and automation that support these operations. o Analytical techniques used for quality assurance. o Sample informatics, including registration and compound quality data.

    View service →