Deep learning for the harmonization of structural MRI scans: a survey

Abstract

Medical imaging datasets for research are frequently collected from multiple imaging centers using different scanners, protocols, and settings. These variations affect data consistency and compatibility across different sources. Image harmonization is a critical step to mitigate the effects of factors like inherent differences between various vendors, hardware upgrades, protocol changes, and scanner calibration drift, as well as to ensure consistent data for medical image processing techniques. Given the critical importance and widespread relevance of this issue, a vast array of image harmonization methodologies have emerged, with deep learning-based approaches driving substantial advancements in recent times. The goal of this review paper is to examine the latest deep learning techniques employed for image harmonization by analyzing cutting-edge architectural approaches in the field of medical image harmonization, evaluating both their strengths and limitations. This paper begins by providing a comprehensive fundamental overview of image harmonization strategies, covering three critical aspects: established imaging datasets, commonly used evaluation metrics, and characteristics of different scanners. Subsequently, this paper analyzes recent structural MRI (Magnetic Resonance Imaging) harmonization techniques based on network architecture, network learning algorithm, network supervision strategy, and network output. The underlying architectures include U-Net, Generative Adversarial Networks (GANs), Variational Autoencoders (VAEs), flow-based generative models, transformer-based approaches, as well as custom-designed network architectures. This paper investigates the effectiveness of Disentangled Representation Learning (DRL) as a pivotal learning algorithm in harmonization. Lastly, the review highlights the primary limitations in harmonization techniques, specifically the lack of comprehensive quantitative comparisons across different methods. The overall aim of this review is to serve as a guide for researchers and practitioners to select appropriate architectures based on their specific conditions and requirements. It also aims to foster discussions around ongoing challenges in the field and shed light on promising future research directions with the potential for significant advancements.

Introduction

Neuroimaging techniques like magnetic resonance imaging (MRI), positron emission tomography (PET) and computerized tomography (CT) play a vital role in studying the brain's structure and function [1,2,3]. These medical imaging modalities provide invaluable insight for diagnosing neurological disorders, understanding brain development, and investigating neurodegenerative processes. In comparison to other neuroimaging modalities, MRI is known for its ability to generate excellent soft tissue contrast, making it instrumental for studying subtle tissue details including the brain or spinal cord [4]. Additionally, it is vital for many differential diagnoses of neurological disorders, including tumors [5], inflammatory conditions [6], and degenerative disorders [7]. MRI utilizes various imaging options and pulse sequences according to clinical needs. These sequences generate images with different contrasts, such as ${T}_{1}$-weighted, ${T}_{2}$-weighted, and PD-weighted (Fig. 1).

Fig. 1

Example images acquired from Guy’s Hospital using a Philips 1.5T system, sourced from the IXI dataset [8]

Variations in scanner hardware, imaging parameters, and acquisition protocols can lead to systematic differences in the appearance and quantitative measures derived from neuroimages. These divergences, if unaddressed, can reduce the statistical power of neuroimaging studies, limit the generalizability of findings across sites, and impede efforts to pool and analyze multi-site datasets. Consequently, there is an increasing demand for harmonizing neuroimaging data to mitigate unwanted inter-site and inter-scanner effects.

In recent years, data-driven harmonization techniques leveraging machine learning have gained significant traction. Deep learning models have demonstrated remarkable ability to capture complex data representations and transformations, making them well-suited for tackling neuroimaging harmonization challenges. By learning from data acquired across multiple sites and scanners, these models can disentangle biologically relevant signals from technical artifacts, enabling the generation of harmonized neuroimages or derived measures. However, compared to other neuroimaging harmonization efforts, one of the factors that make MRI harmonization more complex is that grayscale-based signal intensity in MRI lacks a standardized measure, unlike semi-quantitative measures such as standardized uptake value (SUV) of PET or quantitative measures such as Hounsfield units (HU) of CT [3]. This accentuates the variability in MRI, resulting in differences in contrast-to-noise ratio, temporal resolution, and spatial resolution. These variations have been found to affect the reliability of radiomic analysis [9].

With larger and more varied datasets available in open-source databases, researchers can conduct more in-depth analyses and leverage subtle details within the data. By involving more participants and tracking them over an extended period, researchers gain deeper insights into how the study impacts the subjects, such as the effects of a disease or the aging process [10]. This longitudinal approach also enables a better understanding of the underlying causes of certain diseases, ultimately helping identify optimal treatments, diagnostic methods, and care plans. While larger datasets lead to more accurate results, they also introduce increased variability in how the data are acquired. This acquisition variability issue is almost always present for various reasons. Even at a single site using one scanner, patients may require repeat scans on different MRI scanners if they need additional medical care. Furthermore, the imaging environment itself is prone to changes due to scanner upgrades or replacements occurring over the course of a study [11]. When dealing with images from multiple sites, the inconsistency becomes more pronounced and can lead to domain shift problems.

In the realm of medical imaging, addressing domain shift is crucial for ensuring the reliability and consistency of AI models across different imaging sites and scanners. Domain adaptation (DA) involves fine-tuning a model to perform well on data from a specific target domain, thereby enhancing its accuracy and applicability within that domain. On the other hand, domain generalization (DG) aims to develop models that generalize effectively to unseen domains, without specific access to target domain data during training [12]. An emerging technique, image harmonization, focuses on reducing inter-site variation to facilitate meaningful comparisons and analyses of images across diverse imaging environments.

Domain adaptation enhances model accuracy by injecting domain-specific knowledge into a general AI model. This process typically requires access to target domain data during training, allowing the model to better capture domain-specific features and nuances. However, challenges include the need for additional training data [13], inter-modality heterogeneity [14], and rigorous model evaluation. Despite these challenges, domain adaptation significantly improves model performance within specific domains.

In contrast, domain generalization tackles the broader challenge of developing models that can generalize well across unseen domains. This approach is particularly beneficial in scenarios where acquiring labeled data from every possible domain is impractical. By learning from multiple related domains during training without direct exposure to the target domain, domain generalization reduces the need for extensive labeling efforts and enhances the model's adaptability to new tasks. However, the lack of direct access to target domain data makes domain generalization more challenging than domain adaptation in practical applications [15].

Image harmonization focuses on minimizing inter-site variation in medical imaging, enabling consistent analysis and comparison across different imaging sites and scanners [16]. This technique has shown promising results in standardizing imaging data, thereby improving the reliability of downstream analysis and clinical decision-making. Unlike domain adaptation and generalization, which primarily focus on model training strategies, harmonization directly addresses data variability at the preprocessing stage. This approach simplifies model deployment across diverse clinical settings but may require specialized algorithms tailored to specific imaging modalities.

In the context of MRI harmonization, a "traveling subject" refers to a person or phantom (an object designed to mimic certain properties of human tissues) that is scanned on multiple MRI scanners at different sites. The traveling subject provides a common reference point when being scanned on MRI scanners located at different sites or institutions, which may have different scanner models, field strengths, or acquisition protocols. The data from scanning the traveling subject across sites are used to characterize and correct for scanner-specific variations in the MRI data. The use of traveling subjects is a crucial step in many MRI harmonization pipelines, especially for large multi-site neuroimaging studies, as it provides a way to quantify and mitigate scanner-related effects that could otherwise confound the analysis and interpretation of the pooled dataset.

On a related note, in some cases paired MRI data are used for image harmonization. While the two concepts are related, they are fundamentally different in their application. The purpose of the traveling subject is to directly measure and characterize the scanner-specific variations that need to be harmonized. Paired MRI data, however, refer to having two sets of MRI scans acquired from the same subject, which can be done either within a single scanner (intra-site) or across different scanners (inter-site). Intra-site paired data typically include different imaging modalities (e.g., T1-weighted and T2-weighted images) acquired from the same subject on the same scanner. This intra-site pairing facilitates training by providing complementary information from different modalities. Inter-site paired data, which involve scanning the same subject on both a "source" and a "target" scanner, help in harmonizing data between different scanners by offering corresponding data points under different scanner conditions.

While a traveling subject provides a common reference scanned across all scanners, paired data have separate source and target scans. Also, while traveling subjects are fewer in number but scanned widely across different sites, paired data can involve the full set of study subjects scanned at two sites. Lastly, while traveling subjects directly measure scanner effects, paired data rely on the corresponding subject scans to estimate the scanner-specific transformations required for harmonization. Therefore, while a traveling subject provides a direct measurement of scanner effects, paired data provide a way to estimate and apply those effects to each subject's data during the harmonization process. To provide an example of image harmonization, Fig. 2 demonstrates the traveling subject from six different scanners of the SRPBS Multi-disorder MRI Dataset [17]. As observed, the image contrast varies across scanners, which is known to affect downstream tasks such as tissue segmentation and disease classification.

Fig. 2

Traveling subject from different scanners of the SRPBS Multi-disorder MRI Dataset [12] with detailed acquisition parameters. The contrast has been changed across scanners

In 2022, a comprehensive overview was presented on the various approaches for radiomics harmonization [18]. This review study classified these methods into two categories: image-based harmonization and feature-based harmonization. Image-based harmonization techniques are applied directly to the images before extracting radiomics features, while feature-based harmonization aims to reduce the differences between the extracted features themselves. The choice of these techniques can be constrained by the number of samples available for analysis. The review concluded that, up to that point, none of these harmonization methods had been definitively established as the most effective approach within the analysis process.

Statistical methods have been used for feature-based harmonization [19]. These methods offer fine-grained adjustments, enabling researchers to target specific features for precise correction while maintaining interpretability [20]. The primary advantages of statistical methods include the following:

• Interpretability: Researchers can understand which specific features are being adjusted.

• Targeted Adjustments: Precise correction of predefined features is possible, making these methods ideal for datasets with well-understood relevant features.

However, these methods also have notable limitations:

• Dependence on Predefined Features: They rely on predefined features for correction, potentially limiting their effectiveness in complex datasets where relevant features are poorly defined [21].

• Potential for Overfitting: In scenarios with high variability and noise, statistical methods might overfit to the training data.

Image-based approaches particularly leverage deep learning models for accurate mapping between source and target MRI images [21]. Deep learning has revolutionized the field of medical imaging by enabling automated analysis, diagnosis, and prognosis through the extraction of meaningful features from medical images. Image classification, segmentation [22], image reconstruction [23], and image registration [24] are just a few successful applications of deep learning in medicine. Along these lines, recent development of various neural network architectures that have been proposed specifically for the task of medical image harmonization serve as the primary focus of this review article. The main advantages of image-based approaches are as follows:

• Automation and Scalability: They can automatically learn complex mappings without requiring predefined features.

• Handling Complex Variability: They excel in scenarios with complex and high-dimensional data, effectively capturing intricate patterns and relationships.

However, these methods also face challenges:

• Black-Box Nature: Deep learning models are often criticized for their lack of interpretability, making it difficult to understand how corrections are being made.

• Data Requirements: They typically require large amounts of training data and computational resources, which may not be available in all settings.

These methods can be selected according to the specific characteristics of their datasets and research goals. Statistical methods are preferable when interpretability and targeted feature correction are paramount, especially in well-understood datasets. In contrast, image-based approaches are better suited for complex datasets with poorly defined features and where automated, scalable solutions are needed.

Combining statistical and image-based approaches could potentially yield better performance by leveraging the strengths of both methods. For instance, a hybrid model could use statistical methods to correct well-defined features while employing deep learning to handle more complex, undefined variability. This combination could enhance both the accuracy and interpretability of harmonization efforts. Future research could explore integrated frameworks that synergistically use both approaches, potentially leading to more robust and generalizable harmonization techniques.

Wen et al. provided an examination of image harmonization methods for brain MRI data, with a particular focus on machine learning (ML)-based approaches used in both explicit and implicit ways [25]. They reported that a uniform imaging dataset can be achieved by implementing explicit methods that harmonize intensity values and image-derived metrics, or by using implicit methods to improve the performance of a downstream task. They also noted that traveling subject datasets are crucial for the effective implementation of explicit harmonization, as these enable the machine learning models to avoid learning biased information from the population. However, contemporary traveling subject datasets have limitations in terms of size and issues related to scan–rescan reliability, which can hinder the performance of the ML models. In contrast, implicit methods do not require a traveling subject dataset. Researchers only need to determine the source and reference domains to develop the machine learning algorithm for harmonizing the MRI scans.

Different harmonization solutions, including the image domain and feature domain, have been discussed in another survey [26]. The image domain harmonization encompasses acquisition protocols and data augmentation, while the feature domain category includes statistical normalization, Combat [27], and deep learning. GAN, Neural Style Transfer (NST), and their combination were discussed as deep learning-based harmonization techniques.

Hu et al. investigated both statistical and deep learning methods for harmonization [20]. The study noted that statistical techniques could provide robustness and effectiveness, especially in scenarios with smaller sample sizes or when dealing with confounding factors. Conversely, deep learning models may be better suited to handle the complex nature of image-level data, which poses significant challenges for conventional statistical approaches. A recent book chapter by Zuo et al. [28] reviewed disentangled representation learning methods for MR image harmonization, demonstrating how disentangled representations can be learned through both supervised and unsupervised image-to-image translation techniques.

• Literature search

To narrow down our primary focus, we conducted a comprehensive literature search that met specific inclusion criteria including applying harmonization technique on structural MRI, employing deep learning methods, and in some papers harmonizing images for downstream tasks. The search was organized across the PubMed database using the terms "MRI harmonization" AND "deep learning," "MRI harmonization" AND "structural MRI," and "MRI harmonization" AND "deep learning" AND "structural MRI" through February 2024. This search returned 285 papers and the duplicate papers were excluded in the first phase. The same keywords "disentanglement representation learning for MRI" and "MRI harmonization using transformers" were used for searches on Google Scholar. Studies that were excluded encompass those that (1) were applied on the PET, fMRI, dMRI, or anything other than structural MRI, (2) did not employ a deep learning architecture, and (3) did not belong to the aim and scope of this review according to titles and abstracts. After applying criteria to the literature search and conducting screening, we incorporated a total of 38 papers into our study.

All the papers included in this study were published between 2019 and 2024, with approximately 26% from 2022 and 34% from 2023. The identified studies were analyzed in terms of their network architecture, learning algorithm, network framework, and network output. Based on the deep network architecture, the approaches can be classified into U-Net, GANs, VAEs, flow-based generative models, transformers, and custom Networks. Some networks learn via disentangled representation learning, which involves extracting and separating meaningful features or factors from the MRI data associated with different imaging characteristics. Among the studies, 29% utilized custom networks with or without disentanglement representation, 24% employed GANs, 13% used U-Net, and 13% were based on VAEs. Additionally, two of them relied on transformers, and three utilized flow-based generative models, while the rest employed a combination of two networks.

The majority of articles, about 74%, relied on publicly available datasets; however, 26% used local datasets. In addition to that about 53% of the articles proposed a 3D model or worked with 3D images. It is noteworthy that 66% of the papers employed harmonization solely on T1-weighted images, while the rest also utilized other contrasts. Furthermore, 63% of the studies conducted tests on the harmonized images for downstream tasks, whereas the remainder focused merely on image harmonization. Additionally, 15 of the articles have publicly available source code.

• Motivations

Based on the survey of prior efforts, the motivation behind this review is outlined as follows:

• 1) A thorough, wide-ranging evaluation of deep learning-based methods for harmonizing structural MRI data across various benchmarks is lacking. Such a comprehensive investigation could shed light on the advantages and limitations of existing approaches in this domain.

• 2) While previous review articles have examined various harmonization techniques, they have not covered harmonization strategies that leverage transformers, flow-based generative models, or custom-designed neural network architectures.

• 3) A comprehensive comparison of large-scale brain imaging datasets for training and evaluating harmonization methods has not been conducted.

• 4) There has been a lack of detailed discussion surrounding the evaluation metrics used for assessing MRI harmonization methods, including considerations for scenarios with or without the presence of traveling subject data.

Driven by the aforementioned motivations, the primary goals of this study are to address existing gaps, elucidate the current limitations for a comprehensive comparison of harmonization techniques, and analyze the pros and cons of deep learning architectures for MRI harmonization. As such, our focus is solely on deep learning-based harmonization techniques. The design of networks for MRI harmonization can draw inspiration from networks used for domain adaptation [29] and image-to-image translation [30] in various computer vision applications. However, due to the complex structure of the brain and the intensity levels of MRI images, MRI harmonization networks have fundamental differences from networks in other applications. MRI harmonization aims to preserve the anatomical (content) information of the source image while transforming the contrast (style) to the target domain.

• Contributions

SA performed literature search and wrote first draft of the manuscript. BV reviewed the first draft and conceived the project. SA, HA, JC, NSB, GP, and BV reviewed all subsequent drafts and approved the final manuscript.

• 1. This study provides a categorization of state-of-the-art deep learning-based MRI harmonization techniques based on their architectural design (e.g., GAN, U-Net, VAE), learning algorithms, network frameworks, and network outputs. This systematic classification offers valuable insights into the appropriate design considerations for developing effective harmonization networks.

• 2. This study conducts a comprehensive comparison of well-established large-scale datasets based on their fundamental characteristics, such as the number of participants, age range, target challenges, scanner types, and image modalities. Additionally, it discusses the lack of a dedicated harmonization dataset that addresses specific challenges faced by the medical imaging community.

• 3. This study examines the commonly used evaluation metrics in explicit image harmonization techniques, such as the Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM). It highlights the limitations of these evaluation criteria, pointing out their potential shortcomings in enabling accurate comparisons across different harmonization methods.

The rest of the paper is structured as follows. Sect. "The Overview of Harmonization" introduces harmonization. Sect. "Deep Learning-based Harmonization Taxonomy" categorizes different harmonization techniques. While acknowledging that certain papers may fit into more than one category, they are classified based on their predominant focus. Sect. "Applicability and Limitations of Harmonization" discusses the applicability and limitations of harmonization in MRI neuroimaging in a general context. Sect. "Discussion" analyzes the strengths and weaknesses of the various image harmonization methods. Sect. "Conclusion and Future Direction" provides concluding remarks.

The overview of harmonization

The MRI harmonization is formulated as a transformation that maps images from a source domain ${X}_{s}$ to a target domain ${X}_{t}$. Let ${x}_{s}$ denote an MRI image within the source domain, and ${x}_{t}$ signify its counterpart within the target domain. The objective of MRI harmonization is to learn a mapping function $f$ :

$$f : X_s \to X_t .$$

(1)

The function $f$ should preserve the anatomical content of the source image while aligning its contrast to match that of the target domain. This can be expressed mathematically as follows:

$$x_{t} \, = \,f\left( {x_{s} } \right) + \, \in$$

(2)

where $\epsilon$ represents the residual error introduced during the harmonization process. To learn the mapping function $f$, a deep learning model using adversarial and auxiliary loss functions can be employed to minimize the discrepancy between the distribution of synthesized images ​${x}_{t}$ and real images from the target domain ${X}_{t}$.

There are several key factors to consider when evaluating image harmonization techniques. This section will discuss these factors, including the types of datasets used for training, the scanners employed for image acquisition (if relevant), and the metrics used to assess the success of harmonization approaches. Each of these aspects can influence the effectiveness and suitability of a particular method and can present unique challenges (Fig. 3).

Fig. 3

Overview of Harmonization

Datasets

Pooling MRI data from diverse sources is crucial for high-powered and large-scale brain imaging studies. These sources include different sites, scanners, and acquisition protocols. Fortunately, several large neuroimaging datasets already exist, such as the UK Biobank [31], ABIDE [32], and ADNI [33]. Apart from heterogeneities in data due to scanners and modalities, the following complexities further complicate the harmonization process:

• Disease complexity: While some harmonization methods use healthy brain scans, brain diseases pose additional challenges. Subtle changes in diseased brains are crucial for studies like Alzheimer's disease progression, and harmonization techniques must not obscure this information.

• Segmentation and classification complexity: When diverse images from different scanners are harmonized, it must allow for precise and accurate segmentation (identification of specific brain structures) and classification (grouping images based on disease state).

Biological covariate balance complexity: Ensuring a balanced distribution of biological covariates such as gender and age in training datasets remains important. This is critical if the sample size of the training data is limited. A balanced representation of biological covariates and accounting for their difference helps achieve more generalizable and realistic results.

While Table 1 provides an overview of common datasets used in harmonization research, these datasets present challenges due to their inherent diversity. These large-scale datasets are often collected from multiple sites across different countries, and harmonization studies frequently utilize only a subset of the data or even employ entirely different datasets. This variation in data origin and usage hinders standardization and makes direct statistical comparisons between studies difficult.

Table 1 Large-scale neuroimaging datasets

Scanners

The global estimation for the number of MRI machines was around 36,000 with 2500 machines being manufactured annually [39]. While MRI is a vital tool in medical diagnosis, their inherent limitations can affect image quality and accuracy. These limitations arise from two main sources:

Natural effects

The MRI process itself and the equipment involved are susceptible to natural phenomena that can cause issues. Examples include magnetic field inhomogeneity, gradient non-linearities, and variations in radiofrequency (RF) coil sensitivity. These effects manifest as variations in image intensity, distortions, and artifacts, ultimately impacting image quality and potentially leading to misdiagnosis. Fortunately, manufacturers are constantly working on mitigating these natural effects to improve the reliability and quality of MRI systems.

Acquisition settings

The specific settings used during an MRI scan significantly impact the resulting images. Factors like pulse sequence type, repetition time (TR), echo time (TE), and flip angle contribute to the appearance and accuracy of the scan. Table 2 provides a detailed breakdown of these factors. Selecting and optimizing these settings is crucial to minimize inherent limitations like noise, artifacts, and inconsistencies in image quality. By tailoring these settings to specific diagnostic needs, healthcare professionals can ensure high-quality imaging outcomes.

Table 2 Technical factors that can affect the image quality of MRI scan

Metrics

Accurately assessing the effectiveness of harmonization algorithms is critical. This subsection delves into several essential metrics used for this purpose:

Peak signal-to-noise ratio (PSNR)

This metric measures the ratio between the maximum possible signal (image intensity) and the corrupting noise that affects image quality. Higher PSNR values generally indicate better harmonization and is essential for accurate diagnosis and assessment in clinical settings [46]. Improved PSNR aims to preserve important anatomical details, aiding radiologists in making precise evaluations. Thus, the PSNR ratio is a highly effective quality indicator in evaluating the effectiveness of harmonization architecture.

This ratio is derived from the difference between the original image and the harmonized version. The PSNR is calculated using the following equation:

$$PSNR = 10 \log_{10} \left( {\frac{R^2 }{{MSE}}} \right),$$

(3)

where R demonstrates the maximum fluctuation in the input image data type.

Mean absolute error (MAE)

MAE calculates the average of the absolute differences in pixel intensity values between the original and harmonized images. Lower MAE values indicate that the harmonized image closely resembles the original, preserving essential diagnostic information. This accuracy in intensity values is crucial for tasks in image harmonization, where the goal is to closely approximate the original data. This fidelity is important for various applications, including diagnostic assessments and other clinical evaluations where accurate representation is paramount.

$$MAE = \frac{{\sum_{M,N} \left| {I_1 \left( {m,n} \right) - I_2 \left( {m,n} \right)} \right|}}{M \times N},$$

(4)

where M and N represent the row and column of the input image, respectively.

PSNR and MAE both quantify the difference between the original and harmonized images, but they do so in different ways. PSNR is a logarithmic measure that emphasizes larger differences, making it useful for detecting significant deviations in image quality. MAE, on the other hand, provides a linear measure of average differences, offering a straightforward assessment of overall image fidelity. Together, these metrics provide a comprehensive view of image quality by highlighting both large and small discrepancies.

Structural similarity index measure (SSIM)

SSIM goes beyond just measuring noise levels. It compares the overall structural similarity between two images, considering luminance, contrast, and structure. Clinically, higher SSIM values suggest that the harmonized image retains the structural integrity of the original, which is critical for identifying subtle anatomical changes [47]. This metric helps ensure that the harmonization process does not distort important clinical features, thereby supporting accurate diagnosis and treatment planning.

The SSIM is based on illumination, contrast, and structural terms (Eqs. 6–8). In these equations, $\mu$ and $\sigma$ are local mean and standard deviation, respectively.

$$SSIM\left( {x,y} \right) = \left[ {l\left( {x,y} \right)} \right]^\alpha . \left[ {c\left( {x,y} \right)} \right]^\beta .\left[ {s\left( {x,y} \right)} \right]^\gamma ,$$

(5)

$$l\left( {x,y} \right) = \frac{2\mu_x \mu_y + C_1 }{{\mu_x^2 + \mu_y^2 + C_1 }},$$

(6)

$$c\left( {x,y} \right) = \frac{2\sigma_x \sigma_y + C_2 }{{\sigma_x^2 + \sigma_y^2 + C_2 }},$$

(7)

$$s\left( {x,y} \right) = \frac{{\sigma_{xy} + C_3 }}{\sigma_x \sigma_y + C_3 }.$$

(8)

SSIM complements PSNR and MAE by focusing on structural information rather than just pixel-wise differences. While PSNR and MAE quantify numerical discrepancies, SSIM evaluates how well the harmonized image preserves the structural integrity of the original, considering luminance, contrast, and structural similarity. This combination ensures that both numerical accuracy and structural fidelity are assessed.

In addition to these core metrics, evaluation techniques specific to generative models might also be employed for certain harmonization approaches.

Recent research [48] has highlighted a crucial disconnect between how well harmonization algorithms perform according to traditional metrics and their actual impact on downstream applications. This suggests that common image similarity metrics like PSNR and SSIM might not fully capture the effectiveness of harmonization in improving compatibility across different datasets (cross-domain consistency). As a result, there is a growing need to re-evaluate current metrics to ensure they accurately assess the success of harmonization techniques.

To address this limitation, new assessment methods have been proposed [49]. These methods focus on two key aspects of harmonization:

• Intensity Harmonization: The Wasserstein Distance (WD) is used to measure how well intensity levels are harmonized. It achieves this by calculating the movement of histograms between images, essentially quantifying how similar the intensity distributions become. Clinically, ensuring consistent intensity levels across images from different scanners or protocols can improve the reliability of quantitative measurements, such as volumetric analysis.

• Anatomy Preservation: To evaluate how well anatomical structures are preserved during harmonization, segmentation is performed on both the original and harmonized images. The relative Absolute Volume Difference (rAVD) is then calculated to compare the segmentation results. This provides a measure of how closely the harmonized image retains the anatomical information from the original image. Accurate anatomical preservation ensures that clinical assessments, such as tracking disease progression or planning interventions, are based on reliable data.

Summarizing, WD measures the similarity of intensity distributions, ensuring that the overall intensity levels are harmonized across images. rAVD, on the other hand, evaluates how well anatomical structures are preserved during harmonization. By combining these metrics, we can assess both the consistency of intensity harmonization and the preservation of anatomical details, offering a dual perspective on harmonization effectiveness.

By incorporating these more specific metrics alongside traditional ones, researchers can gain a more comprehensive understanding of how different harmonization algorithms perform and their suitability for real-world applications.

In many real-world applications, "ground truth" data, which represents the perfect or ideal outcome, may not be available. This is especially common in situations involving unpaired datasets, where there is no direct correspondence between elements. Image generation tasks frequently encounter this scenario.

To address this challenge, researchers have developed metrics that assess image quality without relying on ground truth. These metrics offer valuable insights for objectively evaluating model performance:

• Inception Score (IS): A popular metric for assessing the quality and diversity of images generated by models like Generative Adversarial Networks (GANs) [50]. IS considers both the distinctiveness of individual images and the overall variety within the generated set.

• Fréchet Inception Distance (FID): This metric compares the similarity between the distribution of real images and the distribution of generated images in a feature space extracted by a deep learning model [51] (often the Inception network). Lower FID values indicate better alignment between the real and generated data distributions.

• Kernel Inception Distance (KID): Similar to FID, KID measures the discrepancy between the feature representations of real and generated images. However, KID utilizes kernel methods for the comparison. It is primarily used to assess the overall quality of generated images [52].

• Learned Perceptual Image Patch Similarity (LPIPS): This metric goes beyond basic image statistics and leverages a pre-trained deep learning model to assess the perceptual similarity between images. LPIPS considers factors like human visual perception and aims to quantify how similar two images appear to the human eye [53].

By employing these metrics alongside traditional methods, researchers gain a more comprehensive understanding of how models perform in scenarios lacking ground truth data.

While Inception Score (IS), Fréchet Inception Distance (FID), Kernel Inception Distance (KID), and Learned Perceptual Image Patch Similarity (LPIPS) are valuable tools for assessing the quality and diversity of generated images, they may not directly translate to evaluating image harmonization techniques because of the following:

• Differing Goals: Image generation aims to create entirely new, realistic images, whereas harmonization focuses on aligning existing images while preserving their anatomical content. Metrics like IS and FID prioritize diversity and novelty, which might not be desirable in harmonization.

• Focus on Anatomy: Preserving anatomical accuracy is paramount in harmonization. These metrics, however, do not explicitly assess how well anatomical structures are maintained during the process.

However, there might be situations where these metrics could be incorporated into a broader evaluation framework for harmonization, for example, target domain matching. In this scenario, if the harmonization process involves generating synthetic images to match a specific target domain (e.g., MRI scans from a particular scanner model), these metrics could be used to measure the discrepancy between real images from the target domain and the synthetic images produced during harmonization. This could provide insights into how well the harmonized images capture the characteristics of the target domain but would not necessarily address anatomical preservation.

In conclusion, while the mentioned metrics offer valuable insights for image generation, alternative methods are needed to comprehensively assess the success of image harmonization techniques, particularly regarding anatomical fidelity.

Lastly, LPIPS measures the distance between feature representations extracted from pre-trained deep neural networks, like VGG or ResNet. These feature representations capture high-level perceptual qualities of the images, such as textures, shapes, and structures. Unlike metrics like IS or FID, LPIPS considers these learned features, potentially making it a valuable tool for image harmonization. While LPIPS was not specifically designed for harmonization, it offers a unique advantage: assessing the perceptual quality and fidelity of harmonized images. Specifically, it reports on the similarity between the harmonized images and the images from the target domain in terms of human perception. Further research is needed to determine its full effectiveness for harmonization evaluation in various contexts.

These metrics offer insights into the quality and diversity of harmonized images. Clinically, they can be useful in scenarios where harmonization involves generating synthetic images to match a target domain. For instance, when harmonizing images to a specific scanner's characteristics, these metrics help ensure that the generated images align well with the clinical standards of the target domain, thus facilitating consistency in diagnostic practices. IS measures the quality and diversity of generated images, while FID and KID compare the distribution of real and generated images in a feature space. LPIPS evaluates perceptual similarity based on high-level features. When used in harmonization tasks, these metrics can help assess how well the harmonized images match the target domain characteristics, particularly in terms of perceptual and feature-level fidelity.

In addition to the mentioned metrics, it is crucial to evaluate not only the technical aspects of the harmonization process but also its practical utility in real-world applications. It is important to consider how well the harmonized MRI data perform in tasks such as disease classification, age estimation, and ROI segmentation. For instance, ImUnity model [54] assessed the classification ability to identify individuals with ASD (Autism Spectrum Disorder) within the ABIDE database, both before and after the harmonization process. In [55] the improvement of Alzheimer's disease classification was reported after applying a harmonization strategy. In another study [56], the segmentation of the thalamus from various MR image modalities was performed, and the impact of harmonization on the segmentation algorithm was investigated.

Common metrics used to evaluate MRI harmonization methods for downstream tasks like segmentation and classification include Dice Similarity Coefficient, Jaccard Index, Accuracy, Precision, and Recall. Other metrics such as Hausdorff Distance, F1-score, AUC-ROC, and Sensitivity are also employed [57]. These metrics offer quantitative measures of the performance of harmonization methods in tasks like anatomical segmentation and disease classification.

Deep learning-based harmonization taxonomy

Designing a successful harmonization network hinges on four critical elements:

• 1. Data Availability: The type of data available for training, whether paired (corresponding images from source and target domains) or unpaired (images from each domain without direct matches), significantly impacts the design choices

• 2. Loss Functions: These functions mathematically quantify the errors made by the network during training. The specific loss function chosen guides the network toward achieving the desired harmonization goals.

• 3. Backbone Architecture: The underlying architecture of the neural network serves as the foundation for learning image representations. Different architectures offer varying capabilities for feature extraction and image transformation.

• 4. Learning Procedure: The optimization algorithm used to train the network plays a crucial role in its effectiveness. This includes techniques for adjusting network weights and parameters to minimize errors.

The following sections delve into a systematic categorization of deep learning-based harmonization methods. This categorization is based on these four key aspects:

• Network Architecture: The underlying structure of the neural network.

• Network Learning Algorithm: The specific optimization technique used for training.

• Network Supervision Strategy: The approach used to guide the training process of the neural network.

• Network Output: The form of the output generated by the network (e.g., harmonized image, segmentation map).

Figure 4 provides a visual representation of this proposed classification scheme, highlighting the different aspects considered.

Fig. 4

Taxonomy of deep learning-based MRI harmonization approaches

Network architecture

The choice of network architecture is critical for MRI harmonization because it dictates the model's ability to learn and represent the complex relationships between images acquired from different sources or scanners. Different architectures offer varying degrees of complexity and flexibility, which ultimately influence their performance in harmonizing MRI data. Figure 5 provides a historical timeline illustrating the evolution of deep learning networks for MRI harmonization in recent years.

Fig. 5

Timeline of deep harmonization methods

To better understand these methods, we can categorize them based on the underlying network architecture they employ. Some of the commonly used architectures (Fig. 6) in MRI harmonization include U-Net, GANs, VAEs, Flow-based Generative Models, Transformers, and Custom Networks (approaches are not exclusively in one category and are a combination of several networks).

Fig. 6

General diagram for deep learning-based medical image harmonization. A U-Net-based methods architecture. B GAN-based method architecture. C CycleGAN-based method architecture. D VAE-based method architecture. E. Flow-based generative model. F Transformers-based architecture

U-Net

The U-Net architecture has been shown to be successful in segmenting [58] and synthesizing medical images [59]. The U-Net architecture has provided good results in dealing with large and diverse datasets in medical imaging. Due to the skip connections, it effectively retains the finer details from the initial images and has demonstrated strong performance in image-to-image translation [60] (Fig. 6A). One notable advantage of employing U-Net is its ability to enhance data with elastic deformation. It also can extract a large number of feature channels in upsampling. However, an inherent limitation is its comprehensive downsampling, which may result in the loss of spatial information [61].

On the basis of U-Net architecture, a supervised contrast harmonization has been introduced which is called DeepHarmony [62]. An overlap cohort was provided through two different protocols in order to get training data. This technique transforms images from protocols to generate harmonized ones that imitate the contrast of the target protocol. DeepHarmony is trained in two ways: one-to-one and many-to-one. Both of them require four separate networks. The former uses a single contrast from the source protocol and produces an image with the corresponding contrast from the target protocol. The latter employs four of the input contrasts including T1-weighted, FLAIR, PD-weighted, and T2-weighted from the source protocol to generate an output contrast from the target protocol. According to the result, this approach improves the result compared with one-to-one, but it needs further parameters. This approach needs training data of paired traveling subjects. In large-scale studies, it is hard to acquire.

Bottani et al. [63] utilized three architectures based on 3D U-Net to synthesize T1w non-contrast enhancement (T1w-nce) from T1w contrast enhancement (T1w-ce). These modifications included a version with added residual connections referred to as Res-U-Net, a version with incorporated attention mechanisms called Att-U-Net, and a version incorporating both transformer and convolutional layers known as Trans-U-Net. These models were employed as standalone generators and also incorporated into a conditional GAN setup, along with the addition of a patch-based discriminator. Although the models offered a degree of interpretability and provided promising results in brain image segmentation, there is a limitation to creating paired T1w-nce and T1w-ce due to the time and cost constraints. In another study [64], a U-Net model was developed to learn the non-linear transformation from the contrast of a source image to that of a target image across three MRI contrasts. The training and validation have been accomplished using 2D paired MR images.

The U-Net architecture synthesizes images at the pixel level using paired data, necessitating precise image coregistration for effective model training. Consequently, inadequate alignment of paired MR images could result in the loss of certain brain structure information in the generated images.

Generative adversarial network (GAN)

Generative adversarial networks are an approach to generative modeling using deep learning methods. Such architecture can be considered as image-to-image translation, which generates an image in an unsupervised learning task. GANs have attracted enormous interest in image translation, typically to generate new images from existing ones. The incorporation of a generator and a discriminator creates a GAN model (Fig. 6B).

A CycleGAN [65], which is widely used in MRI harmonization, stands as a robust deep learning framework facilitating image-to-image translation without the necessity of paired training data. It is made up of two GANs, including two discriminators and two generators (Fig. 6C). The objective of the model is to comprehend the attributes of the target domain and produce novel images from the source domain that exhibit these attributes. CycleGAN offers several key advantages over other image-to-image translation models. It excels in terms of accuracy by utilizing unpaired data, thus delivering superior results without the need for an extensive collection of paired training images. Its robustness to domain shifts in the data makes it versatile, allowing it to perform well with input images from various domains, enabling a wide range of translation tasks. Moreover, CycleGAN can generate high-quality images with smaller datasets, making it particularly valuable for tasks with limited training data, such as medical image-to-image harmonization [66, 67].

In [68], an unsupervised image-to-image canonical mapping based on CycleGAN was learned from a diverse dataset to a reference domain. This approach was evaluated on brain age prediction and schizophrenia classification to show how can mitigate confounding data variation while retaining semantic information. A Maximum Classifier Discrepancy Generative Adversarial Network (MCD-GAN) was introduced [69], leveraging the benefits of both generative models and maximum discrepancy theory. Komandur et al. [66] employed a 3D CycleGAN to harmonize brain MRI data from diverse sources. They concluded that GAN-harmonized data yield higher accuracy compared to raw data for the age prediction task.

Training the CycleGAN can be time consuming, especially when dealing with a large number of training images. Overfitting is a concern with CycleGAN, possibly resulting in inferior outcomes on unseen data. Interpretability can be a hurdle with CycleGAN, making it challenging to comprehend the rationale behind the model's generated results. To address infant neuroimaging datasets harmonization, S2SGAN (Surface-to-Surface GAN) was introduced [70]. This method combines the spherical U-Net and the CycleGAN. The presented cycle-consistent adversarial networks are based on a spherical cortical surface for harmonizing cortical thickness maps between different scanners.

The majority of unsupervised approaches are unable to differentiate between variability caused by image acquisition and that originating from population differences across different sites. As a result, these methods necessitate datasets to include subjects or patient groups with comparable clinical or demographic profiles. Deep learning frameworks have shown success in dealing with image translation by breaking it down into basic content (e.g., line contours and orientation) and complex style (primarily color and texture) [71]. This framework is known by the “Disentangled Representation” term (more details are provided in part 3–2). All images with the same domain have the same content space but the style can vary. This allows the framework to make changes to the style while maintaining the original content. The aim is to maintain consistency in content but adjust the image style. This approach has been used to produce highly realistic translation results.

In [72], cross-site MRI image harmonization has been considered a style transfer problem instead of a domain transfer problem to overcome the need for the datasets to include patient groups or subjects with homogeneous demographic information. The proposed approach tries to overcome the limitation of the statistical method [73]. Some statistical methods need certain clinical or demographic characteristics of subjects within the dataset to control the acquisition-based variance. This method is capable of harmonizing MRI images without prior knowledge of their scan/site labels and harmonized by infusing style information derived from a single reference image.

The StarGAN is a version of GAN that enables Image-to-Image GANs to perform mapping across more than two domains using a single generator and acquire shared features applicable to all domains [74], whereas conventional models need multiple generators. However, it has limited ability to capture minor feature variations. The StarGAN v2 was utilized to process various datasets using canonical mapping from different sites to a reference domain [75]. By doing so, they reduced the impact of site-based variance while preserving the meaning provided by the input data.

A 3D model named Image Generation with Unified Adversarial Networks (IGUANe) was introduced [76] that benefits domain translation and style transfer methods to harmonize multicenter brain MR images. It extends the CycleGAN architecture by integrating multiple domains for training using a many-to-one approach.

Addressing hallucinations in GANs

Hallucinations refer to the generation of artificial structures in the output images that do not correspond to real anatomical features. This is particularly problematic in medical applications, where the authenticity of every detail is crucial for accurate diagnosis and treatment planning.

The causes of hallucinations in GANs are multifaceted [77]. One primary cause is the imbalance between the generator and the discriminator during training, where the generator may learn to produce plausible but incorrect details to fool the discriminator. Another cause is the lack of sufficient and diverse training data, which can lead to overfitting and the generation of unrealistic artifacts. Additionally, the inherent randomness in GANs can introduce noise that manifests as hallucinations.

To mitigate hallucinations, several strategies have been suggested in the literature. One approach is to enhance the quality and quantity of training data [78]. Techniques such as data augmentation can also help in this regard. Another method is to improve the training process through techniques like progressive growing of GANs [79], where the model is trained on low-resolution images initially and progressively moves to higher resolutions, allowing for more stable training and better-quality outputs.

Regularization techniques, such as spectral normalization and gradient penalty [80], can also help by stabilizing the training dynamics and reducing the likelihood of hallucinations. Additionally, incorporating domain-specific knowledge through the use of hybrid models that combine GANs with traditional image processing techniques or other deep learning models can provide more reliable outputs. By integrating these strategies, future work can aim to reduce the incidence of hallucinations in GAN-generated images, thereby enhancing their clinical applicability and reliability.

Variational autoencoders (VAEs)

The GAN networks and VAE are two of the most popular AI image generators. The VAEs consist of two main architectures: encoders and decoders (Fig. 6D). The encoder learns and encodes the representation of input data and maps it to the latent space. The decoder converts the latent space to get back the original data [81]. According to the comparative study regarding anomaly segmentation on brain MRI images, GAN-based models are recognized for their ability to generate ultra-realistic and sharp images [82]. Meanwhile, AutoEncoders are known for their propensity to produce blurry reconstructions.

Torbati et al. proposed a multi-scanner harmonization framework [83]. This encoder–decoder architecture maps the MRIs from multi-scanners to the latent space and then maps the latent embedding to the harmonized image space. It considers two training steps to preserving the anatomical structure: (1) the harmonized images and input image should be as similar as possible and the variance of embeddings across the scanners should be minimized; (2) ensuring that the output images remain similar across scanners. This step helps maintain uniformity and consistency between various scans.

One-shot learning learns from limited data and has shown significant results in many tasks of medical imaging [84,85,86]. Based on VAEs, a one-shot learning method was proposed for harmonization across imaging locations [49]. During testing, the architecture utilizes an image from a clinical site to create an image that aligns with the intensity scale of the cooperating sites. In another study, a zero-shot learning framework using style-blind autoencoders was introduced [87]. The network was trained to recognize and extract essential content information exclusively. Consequently, the trained network demonstrated the capability for zero-shot harmonization by discarding unknown scanner-dependent contrast information.

The architectures based on the combination of GAN networks and VAEs have been presented in multiple studies. Cackowski et al. introduced ImUnity [54]. To decrease the effect of the scanner or site identity on the training results, the generator (VAE) was equipped with a bias learning module connected to the bottleneck. Additionally, a biological preservation module was proposed to maintain pertinent biological information within the latent space representation.

Flow-based generative model

A flow-based generative model is a type of generative model that transforms a simple input distribution into a more complex data distribution using a series of invertible transformations called flows (Fig. 6E). These models offer exact likelihood evaluation, making them suitable for tasks like density estimation. They are flexible, scalable, and can handle high-dimensional data, making them applicable to various tasks such as image generation and denoising.

Recently, BlindHarmony [88] was introduced as a solution for blind harmonization, where a flow-based blind MR image harmonization framework was developed. BlindHarmony utilized only the target domain dataset during training. The objective is to discover a harmonized image that retains the anatomical structure and contrast of the input source domain image while ensuring a high likelihood in the flow model, thus facilitating harmonization for the target domain by leveraging the invertibility of flow models. Beizaee et al. proposed an unsupervised MR harmonization method based on normalizing flow [89]. Within this framework, a shallow harmonizer network was trained to restore images of the source domain from their augmented counterparts. Subsequently, a normalizing flow network was trained to understand the distribution of the source domain. Ultimately, during testing, modifications were made to the harmonizer network so that the resulting images aligned with the distribution learned by the normalizing flow model of the source domain. In another study, a causal flow-based approach was proposed to address the issue of varying feature distributions in multi-site data utilized for Parkinson's disease classification [90].

Flow-based models are inherently invertible, allowing for bidirectional mapping between domains without loss of information and the transformation process is interpretable so facilitates a better understanding of the harmonization process. However, compared to other generative models, flow-based methods are relatively newer in the field of MRI harmonization, leading to fewer established techniques and benchmarks.

Transformers

Recently, transformers with attention mechanisms (Fig. 6F) have gained promising performance in medical image processing [91] and image-to-image translation [92]. Yao et al. [93] employed two attention-based image-to-image translation frameworks, Morph-UGATIT and QS-Attn [94] for MRI harmonization. The effectiveness of these harmonization strategies was evaluated and compared to the conventional CycleGAN by performing a subcortical segmentation task on a heterogeneous dataset acquired at 1.5T and 3T. Among the frameworks assessed, QS-Attn stands out with the most optimal performance. Morph-UGATIT shows comparable performance to QS-Attn and exhibits enhancements in most subcortical regions compared to the CycleGAN model. They concluded that attention-based harmonization techniques demonstrate notable improvements over the baseline frameworks, especially when combined with diverse downstream tasks like segmentation. In [95] two transformer encoders were introduced to extract both style and content information from MR images, and two decoders were utilized to generate harmonized image patches. Additionally, the impact of changes in image resolution on position encoding was addressed. To capture semantic information in images of varying scales, a content-aware positional encoding scheme method was employed, effectively accommodating images of different sizes.

Custom networks

The custom-designed networks are characterized by architectures that, while sharing similarities with U-Net, GANs, VAEs, flow-based generative models, and transformer-based approaches, include distinct components and configurations tailored to address specific challenges in this research domain. These architectures utilize elements that diverge from typical implementations of the mentioned architectures. They incorporate unique configurations of convolutional layers, pooling layers, domain classifiers, and specially designed blocks that may not fit into these established categories.

Inspired by the adversarial framework and domain adaptation techniques, a harmonization approach was introduced that can be effective for classification, regression, and segmentation tasks while employing two diverse network architectures [96]. Image harmonization can be considered a multi-source joint domain adaptation problem. This approach tries to produce shared feature representations that are invariant to the acquisition scanner while still completing the main task of interest across scanners and acquisition protocols with minimum performance compromise.

An attention-guided domain adaptation was introduced for multi-site MRI harmonization and was applied to automated brain disorder identification [97]. In this framework, the attention discovery and domain transfer modules were defined to automatically pinpoint discriminative dementia-related regions in each whole-brain MRI scan and facilitate knowledge transfer between the source and target domains, respectively. Wolleb et al. [98] introduced a constraint in the latent space of an encoder–classifier network to ignore scanner-related characteristics.

Network learning algorithm

The learning strategies for MRI harmonization has been categorized into two main groups: disentangled learning (DL) methods and non-disentangled learning methods (non-DLs).

• 1. Disentangled Learning (DL) Methods: Disentangled learning methods refer to approaches where the neural network or algorithm is explicitly designed to learn separate and interpretable factors or features from the input data. In the context of MRI harmonization,

  • o DL methods aim to disentangle latent factors such as imaging artifacts, variations in acquisition protocols, tissue types, and other confounding factors that contribute to variability in MRI scans.

  • o These methods typically employ architectures such as VAEs, adversarial training techniques, or other models with explicit mechanisms to learn invariant representations across different datasets.

  • o The goal of DL methods is to improve the robustness and generalization of MRI harmonization by separating out and modeling the underlying factors of variability.

• 2. Non-Disentangled Learning Methods (non-DLs): Non-disentangled learning methods, in contrast, do not prioritize the disentanglement of underlying factors in the input data:

  • o These methods may include traditional neural networks, regression-based models, or simpler machine learning algorithms.

  • o They focus on direct mapping from input (MRI scans with variability) to output (harmonized MRI scans) without explicitly modeling or separating out the distinct factors contributing to variability.

  • o While effective in certain scenarios, non-DL methods may be less robust to dataset variations and might not generalize as well across different MRI datasets with varying acquisition conditions.

Studies have shown that inverting the magnetic resonance imaging signal equation produces an encoded image with disentangled contrast data and contrast-invariant anatomical data. This means the image can be separated into two distinct parts that are denoted by $\beta$ and $\theta$ for anatomical map and contrast, respectively. Figure 7 demonstrates the disentanglement representation learning diagram which uses encoder–decoder architecture.

Fig. 7

Disentanglement representation learning diagram

Dewey et al. proposed an approach that includes two sub-networks [99]. The first one is the encoder and the second one is the decoder. These two sub-networks are connected by the latent space which contains disentangled contrast data and contrast-invariant anatomical data. This encoder and decoder are based on U-Net architecture. For training the network, the T1-weighted and T2-weighted images of the same anatomy using different scanning protocols. It is expected that the network will produce the same $\beta$ value for the same anatomies and equal $\theta$ for images generated from scanners with the same protocol. This approach utilizes the multiple contrast magnetic resonance (MR) images acquired within each site. These intra-site paired data can be found in the same session. But, relying on this technique alone will not provide a globally disentangled latent space. Zuo et al. introduced a similar architecture which is called CALAMITI [100] based on information bottleneck theory. The algorithm learns a global latent space of anatomical and contrast information and it can be adapted to a new testing site using only the data collected at the new site. This architecture requires paired images from the same site during training which might limit certain applications, especially in cases where obtaining multi-contrast images is not feasible. Zuo et al. introduced a method that uses a single MR modality [101]. The inputs of the encoder–decoder are two slices from different orientations of the same 3D volume instead of paired images. Additionally, they defined a new information-based metric for evaluating disentanglement.

According to the inspiration of advancements in multi-domain image translation, Multiple-site Unsupervised Representation Disentanglement (MURD) was introduced [102]. The harmonized images were produced using combining the content of the original image with styles from a specific site or a generator. The style generator enables the generating of multiple appearances concerning natural style variations associated with each site. In [103], another work based on disentangled representation was introduced that disengaged the image into content and scanner-specific space. This method was evaluated on healthy controls and multiple sclerosis (MS) cohorts. Zhao et al. developed a deep learning model to harmonize the multi-site cortical data using a surface-based autoencoder [104]. The encoded cortical features were subsequently decomposed into components related to site-specific characteristics and those unrelated to site effects. An adversarial strategy was employed to promote the disentanglement of these components. Subsequently, the decoding of the site-unrelated features, combined with other site-related features, facilitates the generation of mappings across different sites.

A disentangled latent energy-based style translation (DLEST) framework was introduced [105] in order to harmonize image-level structural MRI. The proposed model disentangles site-specific style translation and site-invariant image generation through the utilization of an energy-based model and a latent autoencoder.

Harmonization with Attention-based Contrast, Anatomy, and Artifact Awareness (HACA3) was suggested to overcome some drawbacks of synthetic-based disentanglement [106]. The previous methods to harmonize MR images are limited by their reliance on assumptions that contrast images from the same subject share the same anatomy. These assumptions are doubtful as different contrasts are aimed at highlighting distinct anatomical features. Moreover, these methods require a fixed set of images for training, which is often limited to T1-weighted and T2-weighted data. Finally, the existing methods are sensitive to artifact images and other image artifacts, making them less useful in practical applications.

Utilizing DRL's advantages, it becomes possible to learn and align independent factors within generation objectives with the latent representation through disentanglement. Consequently, this enables effective control over the generation process [107].

In contrast, non-disentangled representations encapsulate multiple factors of variation in a more intertwined manner, making it challenging to isolate individual factors and understand their influence on the harmonization process. While non-disentangled approaches may offer simplicity and computational efficiency, they often lack the interpretability and robustness necessary for reliable MRI harmonization across diverse datasets.

Network supervision strategy

Broadly, harmonization techniques can be classified into major categories, including supervised, unsupervised, semi-supervised, and self-supervised approaches. Supervised harmonization methods [62, 108] are employed to harmonize images from different scanners/sites using a cross-domain dataset. These methods require a group of subjects to be scanned in both domains. This arrangement provides the training and validation data that the model requires. Due to the logistics and costs of acquiring data, gathering cross-domain data is uncommon in practice. Additionally, cross-domain data are limited. Typically, data from multiple domains are available without cross-domain data. This necessitates an unsupervised harmonization method [66, 89, 109], which requires a training method without data from the same subjects from multiple domains.

Semi-supervised approaches [110], on the other hand, are trained using a dataset containing under-sampled acquisitions of both source and target contrasts from MRI scans. In contrast, self-supervised approaches in the realm of harmonization enable models to learn from the inherent structure of the data itself, eliminating the need for external labels. However, it is important to note that many methods introduced in the field of harmonization are categorized under unsupervised techniques.

Network output

In MRI harmonization, methods can be categorized into two categories based on their network output: direct and indirect. In the direct network output category, methods focus on predicting the target image directly from the reference image. Deep learning models within this category are specifically trained for harmonizing data, allowing for straightforward evaluation by a radiologist. Conversely, methods in the indirect network output category involve training models on a downstream task, such as classification, registration, segmentation, or age prediction. In this category, the harmonization process occurs implicitly through optimization during training, resulting in harmonized data that remain concealed from direct observation. Figure 8 illustrates the general diagram of direct and indirect harmonization.

Fig. 8

Classifying Harmonization according to Network Output

There are some papers in the literature that consider harmonization for downstream tasks. Grigorescu et al. [111] explored two unsupervised domain adaptation techniques, seeking the optimal solution for tissue segmentation maps using T2-weighted magnetic resonance imaging data from an unseen neonatal population born preterm. In [56], a 3D U-Net architecture was presented to segment the thalamus from multiple MR image modalities, and the effect of harmonization on the segmentation algorithm was investigated. Tor-Diez et al. [112] used an unpaired image-to-image translation strategy based on adversarial networks and supervised segmentation for the anterior visual pathway. They concluded that harmonization can improve the segmentation results significantly. In another study, with the aim of boosting Alzheimer's disease classification, the Attention-Guided Generative Adversarial Network (AG-GAN) was used for data harmonization [55]. Komandur et al. [66] proposed a CycleGAN-based harmonization for improving the results of age estimation. According to the assumption that neglecting downstream applications during harmonization can hinder overall performance, the goal-specific harmonization framework was proposed [113]. This VAE-based architecture utilizes downstream application performance to regulate the harmonization procedure. They concluded that while this approach enhances downstream performance, it may also limit generalization to new downstream applications, potentially necessitating repetition of the training procedure for each one.

In indirect approaches, according to downstream tasks, different objective functions for clustering, classification, or regression can be defined, so they need a diverse range of learning procedures, parameters, and optimization algorithms. Therefore, when addressing a different task, the harmonization technique needs to be initiated anew.

The details regarding the papers' information are presented in Table 3. Since the datasets and evaluation metrics are different the comprehensive comparison is limited. The variation in scanners and number of participants, healthy and patient cases, and the investigated contrasts can affect the result and applicability.

Table 3 Details of harmonization techniques that were discussed in this review article. DRL indicates Disentanglement Representation Learning

Applicability and limitations of harmonization

Harmonization techniques in MRI neuroimaging are essential for mitigating scanner-related variability and site effects, allowing for more reliable and comparable results across different studies and cohorts. These methods enhance the statistical power of studies by increasing the effective sample size and facilitating meta-analyses and multi-site collaborations. Harmonization is particularly beneficial in large-scale studies involving data collected from different scanners, protocols, and populations.

Despite its advantages, harmonization is not without limitations. One major concern is that harmonization algorithms may introduce bias if not properly validated across different datasets. The effectiveness of harmonization can vary depending on the specific characteristics of the datasets, including differences in scanner types, imaging protocols, and the populations being studied. Additionally, harmonization processes might inadvertently remove or obscure biologically relevant variations that are not related to scanner differences.

Another limitation is the complexity and computational cost associated with advanced harmonization techniques. Implementing these methods often requires significant expertise and resources, which may not be readily available in all research settings. Furthermore, the choice of harmonization method can impact the results, necessitating careful consideration and validation of the chosen approach.

Neuroimaging analysis should not always use harmonization, especially in scenarios where the primary goal is to investigate scanner-specific effects or when studying the inherent variability between different imaging systems. In such cases, harmonization could mask the very differences that are of interest. Additionally, in single-site studies with consistent imaging protocols, the need for harmonization may be minimal or unnecessary.

Harmonization might negatively impact results if applied inappropriately. For instance, over-harmonization can lead to the loss of important biological signals, resulting in reduced sensitivity to detect true effects. It is crucial to balance the removal of unwanted scanner-related variance with the preservation of genuine biological variability. Researchers should perform extensive validation to ensure that harmonization does not distort the data in a way that affects the study outcomes.

Discussion

Inconsistent contrast across MRI scans presents a significant hurdle for modern medical image analysis techniques. This becomes particularly evident when using deep learning models trained on specific image types. For example, a segmentation model designed for CT scans might perform poorly on MR images due to the fundamental differences in how these imaging techniques capture the body. While research efforts like cross-domain synthesis [114,115,116,117] have aimed to address these challenges, inconsistencies in contrast remain a persistent issue, even within the realm of MRI scans themselves [28].

It is important to distinguish between image harmonization and cross-domain synthesis, although both techniques address challenges with image variability. While image harmonization aims to align images from different sources (e.g., scanners) while preserving anatomical details and spatial relationships, cross-domain synthesis focuses on generating images in a target domain based on images from a different source domain. While the former is particularly useful when combining datasets from different sources for analysis and it helps ensure consistency and reduces variability, enabling more reliable comparisons, the goal of the latter is to preserve key features like structures or textures, while also creating visually realistic images in the target domain. This technique can be used for data augmentation, domain adaptation, and image enhancement. In essence, harmonization aims to make existing images from different sources more compatible, while cross-domain synthesis aims to create entirely new images within a specific domain. Given this distinction, and our focus on harmonization techniques, we have not explored cross-domain synthesis techniques within this paper.

While most harmonization techniques reviewed here leverage 2D images, there is growing recognition that 3D models offer significant advantages. 3D models hold greater potential for capturing the full complexity of medical image features, potentially leading to improved learning performance. However, 3D models come with significant computational limitations such as increased memory requirements to store and process 3D data and longer training times for deep learning models due to the larger amount of data. To address these challenges, some harmonization approaches employ a hybrid strategy for example, 2D Slices with Multi-Orientation. In this approach, models are trained using 2D axial, coronal, and sagittal slices extracted from each 3D MR volume. Subsequently, these multi-directional 2D slices are then combined into a harmonized 3D volume [106].

A majority of current harmonization methods reported in the literature evaluate their performance primarily on T1-weighted MRI scans. While this is a common starting point, it is important to acknowledge the limitation of limited generalizability, i.e., these methods might not achieve the same level of success with other MRI contrasts, such as T2-weighted, PD-weighted, and T2-FLAIR images.

The preprocessing steps before harmonization approach can affect the harmonization outcome. In many cases, the harmonizing native MRI is an essential step. Subsequent investigation should clarify the influence of harmonization on native MRI and explore how the quality of harmonization can be conditional on the preprocessing procedure employed [73]. By understanding this interplay between preprocessing and harmonization, researchers can develop more robust and effective pipelines for MRI data analysis.

Evaluating and comparing different harmonization techniques presents several obstacles. Firstly, there is a current lack of standardized and comprehensive datasets encompassing a wide variety of MRI contrasts, scanner types, and patient demographics (healthy vs. patient groups). Many studies rely on subsets of data, focusing on scanners with minimal differences or specific patient groups. Secondly, given the importance of harmonization in medical image processing, establishing a reference dataset would be highly beneficial. This dataset should ideally include multiple MRI contrasts, involve data from various patient cohorts, and encompass a diversity of challenges commonly encountered in real-world scenarios. Such a benchmark would facilitate more comprehensive evaluation and comparison of harmonization techniques for downstream tasks. Thirdly, conventional image similarity metrics might not fully capture the effectiveness of harmonization. They may prioritize overall similarity without adequately considering factors like cross-domain consistency (compatibility between data from different sources) and preservation of crucial anatomical details. Re-evaluating harmonization success requires metrics that comprehensively assess these key aspects.

While supervised learning approaches using U-Net convolutional neural networks have shown promise in MRI harmonization, they have limitations. Firstly, supervised methods require paired data, meaning the same patients need to be scanned on multiple scanners. This can be expensive and time consuming to acquire, limiting the applicability of these approaches. Secondly, supervised methods often work best for brain imaging due to the relative homogeneity of the MR signal and the feasibility of performing rigid image registration (aligning images based on anatomical landmarks). Addressing these challenges will be crucial to advance the development and evaluation of effective harmonization techniques for broader applications in medical image analysis.

Accurately assessing the effectiveness of different image harmonization techniques remains a challenge due to two key limitations. Firstly, there is an absence of standardized benchmark datasets encompassing a wide variety of factors hinders comprehensive evaluation and comparison. The current studies often rely on the following:

• o Subsets of data: Some studies use a limited portion of a larger dataset, potentially missing valuable information.

• o Homogeneous data: Some studies focus on data acquired from scanners with minimal differences, limiting the generalizability of findings.

• o Data from specific patient groups: Some datasets might be restricted to healthy or diseased individuals, neglecting the real-world scenario where datasets may include both.

Given the crucial role of harmonization in medical image processing, a robust reference dataset is urgently needed. This dataset should ideally include multiple MRI contrasts (T1-weighted, T2-weighted, etc.), data from diverse patient cohorts (healthy and diseased) and a variety of challenges commonly encountered in real-world settings (e.g., scanner variations, acquisition protocols). Such a comprehensive benchmark would enable researchers to thoroughly evaluate and compare harmonization techniques, ultimately improving their performance in downstream tasks.

Conventional image similarity metrics (like PSNR or SSIM) primarily focus on overall image similarity. While important, they may not fully capture the cross-domain consistency, i.e., how well does the harmonized image align with data from a different source (e.g., another scanner)? They may not also fully provide anatomical preservation, i.e., does the harmonized image retain the crucial anatomical details present in the original image? To address these limitations, a re-evaluation of success metrics is necessary. New metrics should be developed, or existing ones adapted to comprehensively assess these essential aspects of harmonization.

Addressing these limitations in datasets and evaluation methods represents a crucial step toward the development and implementation of next-generation harmonization techniques for broader use in medical image analysis.

While U-Net convolutional neural networks have shown promise in supervised learning approaches for MRI harmonization, they face some limitations such as paired data dependency, applicability constraints such as MR signal homogeneity and rigid image registration. These limitations restrict the broader applicability of supervised U-Net-based approaches for harmonization in medical image analysis.

Generative Adversarial Networks (GANs) have been employed to address harmonization by synthesizing images with a specified contrast, where the “content” from the input image is retained while adjusting the contrast to match that of a target scanner. The CycleGAN utilizes unpaired training data and unsupervised learning, showing promise in harmonization tasks and leading to developments in the prediction of brain age and classification. However, an inherent limitation of GANs is their inability to inherently distinguish content from contrast, potentially resulting in alterations to anatomical details to align more with the target scanner dataset, causing “geometry shifts.” Preserving patient anatomy is crucial for precise diagnosis and treatment. In the absence of structural uniformity, the generated images might lack clinically significant specifics. Additionally, GANs are well known for producing artificial structures that are not present in the initial training data, a phenomenon commonly referred to as “hallucination.”

Variational Autoencoders (VAEs) offer an alternative approach to MRI harmonization that addresses a key limitation of supervised learning: the need for paired data. VAEs can potentially harmonize data across multiple sites without requiring scans from the same subjects at each location. This is achieved by learning a latent representation of the data, which essentially captures the underlying characteristics of the images in a compressed form. The VAE then transforms data from one site into another using this latent space. However, some of the limitations of VAEs blurry reconstructions and challenging latent space interpretability. While VAEs hold promise for multi-site harmonization without paired data, further research is needed to address these limitations and improve the accuracy and detail preservation in the harmonized images.

Disentangled representation learning aims to separate an image's style (contrast) and content (anatomy) into distinct representations. This allows for modifications to the style while preserving the underlying anatomical details. However, in complex MRI data, factors like contrast and anatomical details can be intertwined and challenging to perfectly separate. This can lead to ambiguities in the disentanglement process, resulting in overlapping or mixed representations of the intended factors. Additionally, extending disentangled representation learning to 3D or higher dimensions presents additional challenges. The increased complexity of higher-dimensional spaces makes it more difficult to disentangle the features within them.

Vision Transformers (ViTs) have emerged as a powerful tool in computer vision, demonstrating effectiveness across diverse tasks like segmentation, classification, and image-to-image translation. This versatility stems from their core mechanism, self-attention. Unlike traditional convolutional neural networks, ViTs can directly analyze relationships between any two parts of an image, allowing them to capture long-range dependencies and gain a deeper understanding of the global context. However, ViTs also face some limitations due to need for high-resolution input images for optimal performance and substantial computational memory and processing power for training and inference. These factors can limit the applicability of ViTs in scenarios with limited computational resources or where processing speed is critical.

In reviewing the advancements in deep learning models for MRI harmonization, it is evident that even marginal improvements in image quality metrics can be of significant clinical value. However, these improvements often appear minor when comparing new network architectures. The statistical analysis of these improvements is crucial to determine their true significance. For instance [118], highlights the importance of using rigorous statistical methods such as Analysis of Variance (ANOVA) and Mixed Effects Models (MEM).

Similarly [119], investigated three U-Nets (dense, robust, and anisotropic) for upscaling low-quality MRI images. Despite non-statistically significant differences in basic evaluation metrics, mixed effects statistics illustrated significant differences. This suggests that while the detailed architecture of these U-Nets may not drastically alter the outcomes, the use of robust statistical techniques can reveal critical differences and interactions. These findings underscore the importance of employing comprehensive statistical methods to fully understand and validate the performance of different network configurations.

Furthermore, the application of robust statistical techniques, including cross-validation, paired t-tests [120], Wilcoxon signed-rank tests [121], and bootstrap methods, can enhance the reliability, generalizability, and rigor of findings in deep learning model evaluations. These approaches collectively provide a comprehensive framework for assessing model performance beyond subjective evaluation metrics alone. Thus, future research should prioritize not only advancing novel architectures but also ensuring meticulous statistical validation of performance improvements to substantiate their clinical efficacy.

In parallel with these advancements, the integration of foundation models into the harmonization process holds the potential to further refine image quality and consistency across diverse datasets. Foundation models, which are large-scale, pre-trained deep learning models, have recently attracted significant attention across various deep learning challenges. These models are trained on extensive datasets to enhance generalization, contextual reasoning, and adaptability across different modalities. They can be fine-tuned for new tasks using task-specific prompts without the need for extensive retraining or labeled data. The field of medical imaging is increasingly exploring these models to leverage their advanced capabilities and improve outcomes [122].

While the application of foundation models in image harmonization is still an emerging field, these models offer substantial potential for improving consistency and compatibility across diverse medical imaging datasets. Future research should focus on exploring and optimizing the use of these models, conducting comprehensive quantitative comparisons, and addressing the specific challenges associated with harmonization in medical imaging.

Conclusion and future direction

This review provides a comprehensive overview of state-of-the-art deep learning-based methods for harmonizing Magnetic Resonance Imaging (MRI) scans. We categorized harmonization approaches based on their underlying network architecture, including U-Net, Generative Adversarial Networks (GANs), Variational Autoencoders (VAEs), flow-based models, and transformers. We surveyed current literature on MRI harmonization and report significant progress in harmonization techniques, with improvements observed in downstream tasks that rely on harmonized images.

Despite these advancements, several challenges remain:

• Data Standardization: The wide variety of acquisition parameters across scanners, disease states, and patient demographics (e.g., gender) can affect brain size and pose challenges for harmonization robustness. Developing standardized datasets encompassing these diverse scenarios and incorporating comprehensive evaluations for each challenge would be valuable.

• Evaluation Metrics: Current metrics primarily focus on contrast similarity. Novel metrics are needed to assess how well harmonization techniques address the "shift problem" (differences in image intensity distributions) while preserving crucial anatomical information.

• Multi-Site Harmonization: Current methods often focus on harmonization between two specific sites. Exploring techniques that can handle data from multiple sites would be beneficial.

• Architectural Innovation: Combining the strengths of different network architectures (e.g., U-Net for segmentation and GANs for image generation) could lead to more robust harmonization solutions. Additionally, computational efficiency should be considered, as faster models are more practical for real-world applications.

• Generalizability: Extending harmonization frameworks beyond specific MRI contrasts (T1-weighted, PD-weighted, T2-FLAIR) and even exploring other modalities like PET or CT could be a promising research direction.

By addressing these challenges and exploring new avenues, deep learning has the potential to further revolutionize MRI harmonization, ultimately leading to improved medical diagnosis and treatment planning.