- Open Access
Magnetic microbubble-mediated ultrasound-MRI registration based on robust optical flow model
© Hou et al.; licensee BioMed Central Ltd. 2015
- Published: 9 January 2015
As a dual-modality contrast agent, magnetic microbubbles (MMBs) can not only improve contrast of ultrasound (US) image, but can also serve as a contrast agent of magnetic resonance image (MRI). With the help of MMBs, a new registration method between US image and MRI is presented.
In this method, MMBs were used in both ultrasound and magnetic resonance imaging process to enhance the most important information of interest. In order to reduce the influence of the speckle noise to registration, semi-automatic segmentations of US image and MRI were carried out by using active contour model. After that, a robust optical flow model between US image segmentation (floating image) and MRI segmentation (reference image) was built, and the vector flow field was estimated by using the Coarse-to-fine Gaussian pyramid and graduated non-convexity (GNC) schemes.
Qualitative and quantitative analyses of multiple group comparison experiments showed that registration results using all methods tested in this paper without MMBs were unsatisfactory. On the contrary, the proposed method combined with MMBs led to the best registration results.
The proposed algorithm combined with MMBs contends with larger deformation and performs well not only for local deformation but also for global deformation. The comparison experiments also demonstrated that ultrasound-MRI registration using the above-mentioned method might be a promising method for obtaining more accurate image information.
- Particle Swarm Optimization
- Mutual Information
- Root Mean Square
- Optical Flow
- Active Contour Model
Compared with other medical imaging modalities, ultrasound imaging has been widely used in diagnosis and clinical applications owning to its merits of low-cost, real-time, high safety, and no documented side effects. By using proper contrast agents, the contrast and sensitivity of ultrasound imaging have been greatly improved [1, 2]. However, ultrasound imaging is still limited because of the following reasons. Firstly, ultrasound is reflected very strongly when passing from tissue to gas, and vice versa. Secondly, the method is of limited use in diagnosing fractures because ultrasound does not pass well through bones. Finally, the quality of ultrasound imaging is mediocre when its contrast is lower than that of MRI or computed tomography (CT) [3, 4].
Magnetic resonance imaging is another imaging tool which is non-invasive and capable of providing functional information with high spatial resolution and excellent soft-tissue contrast . In particular, MRI can provide information about blood flow and vessel morphology and identify stenotic arteries for early treatment. Magnetic iron oxide nanoparticles with superparamagnetic property can be used as a powerful contrast agent for MRI to further increase its brightness and contrast. One shortcoming of MRI is that it can not provide real-time motion-related images. MRI can be affected by movement, making it unsuitable for investigating problems such as mouth tumors because coughing or swallowing can make the images less clear.
To sum up, different imaging modalities have their respective advantages and disadvantages in the spatial resolution, and no single imaging modality possesses all the advantages satisfying the need of all clinical applications. In many cases, US image and MRI are complimentary, and both modalities are needed to discern possible pathological changes in tissue . Therefore, it is extremely desirable to fuse the image information of different modes. To fuse US and MRI together, US-MRI registration is required. Due to US image's strong noise, it is a great challenge to register US image with any other modality images. Some studies focused on three-dimensional US-MRI registration or three-dimensional US-CT registration in the field of operation navigation [7–12]. Whether feature-based or voxel-based registration is used, segmentation of multimodality medical images is required. The registration result depends directly on the segmentation results. It is fair to say that US image segmentation is a difficult issue at present.
We have carried out preliminary research on MRI-ultrasound registration based on dual modality contrast agent, namely MMBs, and have obtained some promising results [13, 14]. Those elementary research results depend on how to select the region of interest (ROI) to certain extent, while running the registration code is time-consuming, and selecting ROI is a subjective process. To overcome the above limitations, a more efficient registration algorithm and semi-automatic segmentation of ROIs using MMBs are presented in this paper. With MMBs, the gray value within regions of interest (ROI) of US image and MRI is enhanced, which is very favorable for the segmentation to be performed. The MMBs discussed in the paper are the dual-modality contrast agent with both ultrasound and MRI contrast function. Gas filled microbubbles encapsulated with polymer, lipid or surfactant shells can be used as the most effective contrast agent for ultrasound imaging. Superparamagnetic iron oxide nanoparticles (SPIO) can be used as a powerful contrast agent for MRI. The combination of microbubbles and SPIOs, MMBs, can be used as the contrast agent for both US imaging and magnetic resonance imaging because the MMBs can overcome the shortcomings of magnetic nanoparticles or microbubbles, respectively. That is, the stability of microbubbles can be improved by embeding magnetic nanoparticles into the bubble shells. Moreover, the embedded nanoparticles can be delivered into desired regions under the guidance of magnetic field and can be released when suitable ultrasound exposure is chosen. Cai et al focused on the relationship between the MMBs structure and dual modality imaging, and gave a good overview on magnetic microbubbles for theranostics, including their preparation, imaging contrast agents (diagnostic) and drug delivery (therapeutic) .
Different from , the main contribution of this paper is the introduction of the above-mentioned dual modality contrast agent to multi-modality medical image registration. Using MMBs with the mean diameter of 3.98 μm prepared as described by Yang et al [2, 6, 16], this paper carries out the registration between US image and MRI, and comes to a conclusion that with the use of MMBs, the proposed algorithm performs well not only for global deformation but also for local deformation. The remainder of the paper is organized as follows: the registration method based on robust optical flow model between US image segmentation and MRI segmentation is described in Section 2. Section 3 provides several groups of comparison experiments, and analyzes the experimental results, while Section 4 concludes our paper.
Image segmentation with active contour model
Because all US images are confounded by speckle noise, fully automatic segmentation of US image is currently impossible. Therefore, the focus has been on semi-automatic techniques, particularly active contour models which can detect objects whose boundaries are not necessarily defined by gradient. Active contour models are used to evolve a curve, subject to constraints from a given image under a number of external and internal forces [18–22]. The external forces attract the curve to regions of interest in the image, whereas the internal forces hold the curve smooth. If initialized close to a boundary, the curve deforms to "hug" the boundary along its length, providing further adaptability to noise.
here μ ≥ 0, ν ≥ 0, λ11 > 0, and λ22 > 0 are fixed parameters, I0(x, y) is image intensity at pixel location (x, y), and the constants c1, c2 (depending on C) are the averages of I0 inside C and outside C, respectively. The length of the curve, Length (C), and the area of the region inside C, Area (inside (C)), are two regularizing terms. If the constant μ is larger, then only larger objects are detected, or objects that are grouped together. If it is small, then smaller objects will be detected. We do not want different objects close to each other to be interpreted as a single object. This is the reason we decided to set μ = 0 in Eq. (1). The minimization procedure uses iterative method and differential calculus. After initializing a curve close to the object boundary, the active contour model starts deforming to fit the local minimum so as to move towards the desired boundary and finally settles on it.
Robust optical flow model construction for registration
for (i', j') ∈ N i,j and .
for , its proof is similar with . Coarse-to-fine Gaussian pyramid and graduated non-convexity (GNC) schemes are adopted to estimate (u, v) and . A two stage GNC process is adopted and 3 warping steps per pyramid level are performed. After every warping step, (u, v) are set to be . Finally, are taken as the final flow vector field estimate. The framework of Gaussian pyramid algorithm is shown in Figure 2, and the penalty functions are set according to Sun et al .
Materials and data acquisition
Dual modality contrast agent
MMBs were obtained from Jiangsu Laboratory for Biomaterials and Devices. MMBs can increase magnetic resonance susceptibility, and negatively enhance T2-weighted (T2*WI) imaging signal, namely, decrease the gray value of T2*WI imaging. They can also give strong ultrasound backscattering echo intensity and positively increase the brightness of US image.
Phantom was made from glycerol, agar and water ratio of 3:4:90, and produced by Jiangsu Laboratory for Biomaterials and Devices, in which a "U" shaped silicone tube with external diameter of 9 mm and inner diameter of 7 mm is "vertically" sitting in the agar phantom. Three fatty objects which size is about a diameter of 5 mm were laid in the tube. When US imaging or magnetic resonance imaging with MMBs was carried on, a solution (0.1 g/ml) containing MMBs was injected into the silicone tube; while when US imaging or MRI without MMBs was carried on, purified water was injected into the silicone tube. Regardless of whatever MMBs is used, fluid in the tube remains stable during imaging.
Experimental computing platform
Our algorithm implementation is based on the compatible personal computer installed MATLAB 2008b. In some cases, C++ language compilation system was also needed to perform some functions, so it is necessary to install VISUAL C++ 6.0 (or VISUAL C++ 2000.NET) on the above platform.
The three objects were imaged using the Ultrasonic imaging system of the GE LOGIQ3 PRO scanner (GE Medical System, USA) with a 4 MHz ultrasound transducer used as a transmitter as well as a receiver. B-mode US images were acquired with the instrument parameters (Gn 20; E/A 1/2; DR 78; AO 100%). T2*WI imaging of these objects was carried out with a clinical 0.3 T magnetic resonance imager (AIRIS II, Hitachi Ltd, JAPAN). Images were aquired with a matrix size of 256 × 256, field of view of 20 × 20 cm2, repetition time of 400 ms, section thickness of 4 cm, and echo time of 15 ms. The reasons the usage of 0.3 T magnetic resonance imager was as follows. Firstly, we had carried out multiple group comparison tests of magnetic resonance imaging using 1.0 T or above equipments based on MMBs, and obtained good imaging result. Secondly, now SIEMENS 0.3 T magnetic resonance imager was still used widely especially in underdeveloped area because of magnetic resonance imager's expensive upgrades.
Compared with Figure 3(a), the brighter objects of ROI can be seen distinctly in Figure 3(b). This illustrates qualitatively that MMBs can positively enhance the intensity of US. Similarly, the MRIs of the tube without and with MMBs were shown in Figure 3(c)&(d), respectively. The result indicates that MMBs can significantly reduce T2*WI signal intensity. Certainly, we can quantitatively measure objects brightness of US images and MRIs to determine how much MMBs influenced US imaging and magnetic resonance imaging .
Semi-segmentation of US images and MRIs
US-MRI registration based on the proposed method
Comparison of registration results (see Figures 7 & 8) by different values of parameter λ2 (λ1 = 5, λ3 = 1) based on the proposed method.
λ2 = 1e-4
λ2 = 1e-2
λ2 = 1
λ2 = 1e+2
λ2 = 1e+4
λ2 = 1e-4
λ2 = 1e-2
λ2 = 1
λ2 = 1e+2
λ2 = 1e+4
When the reference and the floating image (or the registered image), are compared, the RMS of the pairwise differences of the two images can serve as a measure how far on average the error is from 0. When RMS is small, the similarity between the two images is greater.
Peak signal-to-noise ratio, often abbreviated PSNR, is an engineering term for the ratio between the maximum possible power of a signal and the power of corrupting noise that affects the fidelity of its representation. This ratio can be used as a quality measurement between the reference and the registered image. The higher the PSNR, the better the quality of the registered image.
COR is a mathematical measure of how much one image can expect to be influenced by changes in another. It is closely related to covariance. If there is no relationship between the two images the COR is very low.
MI is a fundamental concept in information theory, and a measurement about statistical correlation of two random variables. Consider gray values of two images which will be registered as two random variables, when the both images achieve the best registration, MI approaches the maximum. When RMS after registration is smaller than before registration, and three indices (PSNR, COR and MI) after registration is larger than before registration, it is called the normal variation, otherwise called abnormal variation in the following discussion.
As shown in Table 1 we can achieve a better registration result independent of MMBs based on the proposed methods when λ1, λ2 and λ3 are set to 5, 1e+2 and 1, respectively.
Comparison of the registration results (see Figures 9 & 10) based on the proposed method (λ1 = 5, λ2 = 1e+2, λ3 = 1).
the proposed method
US-MRI registration based on the other methods
Comparison of the registration results (see Figures 11 & 12) based on the other methods.
In addition, with MMBs and before registration, Table 2 shows RMS, PSNR, COR and MI are 0.0657, 23.6063, 0.1415 and 0.0398, respectively. After registration, for FFT, PSO and MMI, the above evaluation indexes are improved.
The above quantitative analyses also indicate that FFT is the best method among the other three methods, which is in accordance with the above intuitive observation as shown in Figure 12. It should be noted that the evaluation indexes of the proposed method with MMBs, namely RMS, PSNR, COR and MI, are 0.0435, 27.1892, 0.5303 and 0.0803, while the evaluation indexes of FFT with MMB are 0.0565, 24.9184, 0.3684 and 0.0736. Clearly, the performance evaluation of the registration results using the proposed method are superior to the method with FFT. In summary, compared with the other methods, the proposed method combined with MMB has the best performance.
In this study, MMBs were introduced as a new dual-modality contrast agent into the field of medical imaging. We verified that MMBs can increase the contrast of both US image and MRI, leading to the potential beneficial to registration of US and MR images. Using the same contrast agent for both US image and MRI would not only bring convenience to medical professionals, but also reduced health care cost. Qualitative and quantitative analyses of multiple group comparison experiments showed that registration results using all methods tested in this paper without MMBs were unsatisfactory. On the contrary, the proposed method combined with MMBs led to the best registration results.
Our algorithm implementation was intensity-based and was independent of the metric used. Therefore, it can be adapted to different image modalities. At present, effort to improve algorithms for medical image processing has seen very little progress. Combining novel nanomaterials with algorithm optimization provides a new approach for potential gains in imaging processing.
Our results were encouraging. However, they were still at preliminary stage. Further in vivo studies including toxicological and pathological studies will be necessary before our methods could be implemented in clinical applications.
The Southeast University group (Mo Hou, Shouhua Luo, Fang Yang and Ning Gu) has been publishing in preparation and application of magnetic microbubbles, see website: http://lbmd.seu.edu.cn/research.php
Chen has been doing research in medical image processing, see website: http://cae.nuaa.edu.cn/swyx/webs/teachers/chencx.htm
Tang has been publishing image-based modeling work in recent years, For more information, please visit Tang's website: http://users.wpi.edu/~dtang/.
This work was supported by the National Important Science Research Program of China (Nos.2011CB933503, 2013CB733804), the NSFC (Nos. 31000453, 61127002, 61179035 and 61127002), Foundation for the Author of the National Excellent Doctoral Dissertation of PR China (No. 201259), Research Fund of Young Teachers for the Doctoral Program of Higher Education of China (No. 20100092120038), and the Open Project Program (KF-GN-201107).
Publication of this article was paid with funding from the National Important Science Research Program of China (Grant No. 2013CB733804).
This article has been published as part of BioMedical Engineering OnLine Volume 14 Supplement 1, 2015: Cardiovascular Disease and Vulnerable Plaque Biomechanics. The full contents of the supplement are available online at http://www.biomedical-engineering-online.com/supplements/14/S1
- Ke H, Wang J, Dai Z, Jin Y, Qu E, Xing Z, Guo C, Yue X, Liu J: Gold-nanoshelled microcapsules: a theranostic agent for ultrasound contrast imaging and photo-thermal therapy. Angew Chem Int Ed 2011, 123: 3073–7. 10.1002/ange.201008286View ArticleGoogle Scholar
- Yang F, Li L, Li Y, Chen Z, Wu J, Gu N: Superparamagnetic nanoparticle-inclusion microbubbles for ultrasound contrast agents. Phys Med Biol 2008, 53: 6129–41. 10.1088/0031-9155/53/21/016View ArticleGoogle Scholar
- Park JI, Jagadeensan D, Williams R, Oakden W, Chung S, Stanisz GJ, Kumacheva E: Microbubbles loaded with nanoparticles: a route to multiple imaging modalities. ACS Nano 2010, 4: 6579–86. 10.1021/nn102248gView ArticleGoogle Scholar
- Kiessling F, Gaetjens J, Palmowski M: Application of molecular ultrasound for imaging integrin expression. Theranostics 2011, 1: 127–34.View ArticleGoogle Scholar
- Liu Z, Lammers T, Ehling J, Fokong S, Bornemann J, Kiessling F, Gätjens J: Iron oxide nanoparticle-containing microbubble composites as contrast agents for MR and ultrasound dual-modality imaging. Biomaterials 2011, 32: 6155–63.Google Scholar
- Yang F, Li Y, Chen Z, Zhang Y, Wu J, Gu N: Superparamagnetic iron oxide na-noparticle-embedded encapsulated microbubbles as dual contrast agents of magnetic resonance and ultrasound imaging. Biomaterials 2009, 30: 3882–90. 10.1016/j.biomaterials.2009.03.051View ArticleGoogle Scholar
- Arvanitis CD, Livingstone MS, McDannold N: Combined ultrasound and MR imaging to guide focused ultrasound therapies in the brain. Phys Med Biol 2013, 58: 4749–61. 10.1088/0031-9155/58/14/4749View ArticleGoogle Scholar
- Zhang D, Li Z, Chen K, Xiong J, Zhang X, Wang L: An optical tracker based robot registration and servoing method for ultrasound guided percutaneous renal access. Biomed Eng Online 2013, 12: 47. 10.1186/1475-925X-12-47View ArticleGoogle Scholar
- Hu Y, Ahmed HU, Taylor Z, Allen C, Emberton M, Hawkes D, Barratt D: MR to ultrasound registration for image -guided prostate interventions. Med Image Anal 2012, 16: 687–703. 10.1016/j.media.2010.11.003View ArticleGoogle Scholar
- Kuklisova-Murgasova M, Cifor A, Napolitano R, Papageorghiou A, Gerardine Quaghebeur, Rutherford MA, Hajnal JV, Noble JA, Schnabel JA: Registration of 3D fetal neurosonography and MRI. Med Image Anal 2013, 17: 1137–50. 10.1016/j.media.2013.07.004View ArticleGoogle Scholar
- Lee JD, Huang CH, Yang ST, Chu YH, Shieh YY, Chen JW, Lin KJ: MRI/SPECT-based diagnosis and CT-guided high-intensity focused-ultrasound treatment system in MPTP mouse model of Parkinson's disease. Med Eng Phys 2013, 35: 222–30. 10.1016/j.medengphy.2012.01.006View ArticleGoogle Scholar
- Wein W, Roper B, Navab N: Integrating diagnostic B-mode ultrasonography into CT-based radiation treatment planning. IEEE Trans Med Imaging 2007, 26: 866–79.View ArticleGoogle Scholar
- Hou M, Luo S, Yang F, Ruan X, Gu N: MRI-ultrasound registration based on dual modality contrast agent. (in Chinese). Chin Sci Bull (Chin Ver) 2012, 57: 2119–26. 10.1360/972011-2715View ArticleGoogle Scholar
- Hou M, Luo S, Yang F, Ruan X, Gu N: MRI-ultrasound fusion based on magnetic microbubble contrast agent. (in Chinese). Sci China Inf Sci (Chin Ver) 2013, 43: 1445–59.Google Scholar
- Cai X, Yang F, Gu N: Applications of Magnetic Microbubbles for Theranostics. Theranostics 2012, 2: 103–12. 10.7150/thno.3464View ArticleGoogle Scholar
- Yang F, Zhang M, He W, Chen P, Cai X, Yang L, Gu N, Wu J: Controlled release of Fe3O4na-noparticles in encapsulated microbubbles to tumor cells via sonoporation and associated cellular bioeffects. Small 2011, 7: 902–10. 10.1002/smll.201002185View ArticleGoogle Scholar
- Lin X, Qiu T, Nicolier F, Ruan S: The study of active demons algorithm for deformable image registration. Chin J Biomed Eng 2008, 27: 636–40.Google Scholar
- Kass M, Witkin A, Terzopoulos D: Snakes: active contour models. Int J Comput Vision 1988, 4: 321–31.View ArticleGoogle Scholar
- Chan TF, Vese LA: Active Contours Without Edges. IEEE Trans Image Process 2001, 10: 266–77. 10.1109/83.902291MATHView ArticleGoogle Scholar
- Ge Q, Xiao L, Huang H, Wei Z: An active contour model driven by anisotropic region fitting energy for image segmentation. Digit Signal Process 2013, 23: 238–43. 10.1016/j.dsp.2012.06.007MathSciNetView ArticleGoogle Scholar
- Jiang S, Zhang W, Wang Y, Chen Z: Brain extraction from cerebral MRI volume using a hybrid level set based active contour neighborhood model. Biomed Eng Online 2013, 12: 31. 10.1186/1475-925X-12-31View ArticleGoogle Scholar
- Zheng Q, Dong EQ: Narrow Band Active Contour Model for Local Segmentation of Medical and Texture Images. Acta Automatica Sin 2013, 39: 21–30. 10.1016/S1874-1029(13)60003-8MathSciNetView ArticleGoogle Scholar
- Xiong J, Luo Y, Tang G: An Improved Optical Flow Method for Image Registration with Large-scale Movements. Acta Automatica Sin 2008, 34: 760–64.MATHView ArticleGoogle Scholar
- Horn BK, Schunck BG: Determining optical flow. Artif Intel 1981, 17: 185–203. 10.1016/0004-3702(81)90024-2View ArticleGoogle Scholar
- Sun D, Roth S, Black MJ: A Quantitative Analysis of Current Practices in Optical Flow Estimation and The Principles Behind Them. Technical Report 2010. Brown-CS-10–03Google Scholar
- Gilboa G, Osher S: nonlocal operators with applications to image processing. Multiscale Model Simul 2008, 7: 1005–28.MATHMathSciNetView ArticleGoogle Scholar
- Yang Z, Penczek PA: Cryo-EM image alignment based on nonuniform fast Fourier transform. Ultramicroscopy 2008, 108: 959–69. 10.1016/j.ultramic.2008.03.006View ArticleGoogle Scholar
- Xie H, Hicks N, Keller GR, Huang H, Kreinovich V: An IDL/ENVI implementation of the FFT-based algorithm for automatic image registration. Comput Geosci 2003, 29: 1045–55. 10.1016/S0098-3004(03)00104-3View ArticleGoogle Scholar
- Das A, Bhattacharya M: Affine-based registration of CT and MR modality images of human brain using multiresolution approaches: comparative study on genetic algorithm and particle swarm optimization. Neural Comput Appl 2011, 20: 223–37. 10.1007/s00521-010-0374-8View ArticleGoogle Scholar
- Sandiego CM, Weinzimmer D, Carson RE: Optimization of PET-MR registrations for nonhuman primates using mutual information measures: A Multi-Transform Method (MTM). NeuroImage 2013, 64: 571–81.View ArticleGoogle Scholar
- Gao Z, Gu B, Lin J: Monomodal image registration using mutual information based methods. Image Vision Comput 2008, 26: 164–73. 10.1016/j.imavis.2006.08.002View ArticleGoogle Scholar
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.