Magnetic microbubble-mediated ultrasound-MRI registration based on robust optical flow model

Background As a dual-modality contrast agent, magnetic microbubbles (MMBs) can not only improve contrast of ultrasound (US) image, but can also serve as a contrast agent of magnetic resonance image (MRI). With the help of MMBs, a new registration method between US image and MRI is presented. Methods In this method, MMBs were used in both ultrasound and magnetic resonance imaging process to enhance the most important information of interest. In order to reduce the influence of the speckle noise to registration, semi-automatic segmentations of US image and MRI were carried out by using active contour model. After that, a robust optical flow model between US image segmentation (floating image) and MRI segmentation (reference image) was built, and the vector flow field was estimated by using the Coarse-to-fine Gaussian pyramid and graduated non-convexity (GNC) schemes. Results Qualitative and quantitative analyses of multiple group comparison experiments showed that registration results using all methods tested in this paper without MMBs were unsatisfactory. On the contrary, the proposed method combined with MMBs led to the best registration results. Conclusion The proposed algorithm combined with MMBs contends with larger deformation and performs well not only for local deformation but also for global deformation. The comparison experiments also demonstrated that ultrasound-MRI registration using the above-mentioned method might be a promising method for obtaining more accurate image information.


Background
Compared with other medical imaging modalities, ultrasound imaging has been widely used in diagnosis and clinical applications owning to its merits of low-cost, real-time, high safety, and no documented side effects. By using proper contrast agents, the contrast and sensitivity of ultrasound imaging have been greatly improved [1,2]. However, ultrasound imaging is still limited because of the following reasons. Firstly, ultrasound is reflected very strongly when passing from tissue to gas, and vice versa. Secondly, the method is of limited use in diagnosing fractures because ultrasound does not pass well through bones. Finally, the quality of ultrasound imaging is mediocre when its contrast is lower than that of MRI or computed tomography (CT) [3,4].
Magnetic resonance imaging is another imaging tool which is non-invasive and capable of providing functional information with high spatial resolution and excellent softtissue contrast [5]. In particular, MRI can provide information about blood flow and vessel morphology and identify stenotic arteries for early treatment. Magnetic iron oxide nanoparticles with superparamagnetic property can be used as a powerful contrast agent for MRI to further increase its brightness and contrast. One shortcoming of MRI is that it can not provide real-time motion-related images. MRI can be affected by movement, making it unsuitable for investigating problems such as mouth tumors because coughing or swallowing can make the images less clear.
To sum up, different imaging modalities have their respective advantages and disadvantages in the spatial resolution, and no single imaging modality possesses all the advantages satisfying the need of all clinical applications. In many cases, US image and MRI are complimentary, and both modalities are needed to discern possible pathological changes in tissue [6]. Therefore, it is extremely desirable to fuse the image information of different modes. To fuse US and MRI together, US-MRI registration is required. Due to US image's strong noise, it is a great challenge to register US image with any other modality images. Some studies focused on three-dimensional US-MRI registration or three-dimensional US-CT registration in the field of operation navigation [7][8][9][10][11][12]. Whether feature-based or voxel-based registration is used, segmentation of multimodality medical images is required. The registration result depends directly on the segmentation results. It is fair to say that US image segmentation is a difficult issue at present.
We have carried out preliminary research on MRI-ultrasound registration based on dual modality contrast agent, namely MMBs, and have obtained some promising results [13,14]. Those elementary research results depend on how to select the region of interest (ROI) to certain extent, while running the registration code is time-consuming, and selecting ROI is a subjective process. To overcome the above limitations, a more efficient registration algorithm and semi-automatic segmentation of ROIs using MMBs are presented in this paper. With MMBs, the gray value within regions of interest (ROI) of US image and MRI is enhanced, which is very favorable for the segmentation to be performed. The MMBs discussed in the paper are the dual-modality contrast agent with both ultrasound and MRI contrast function. Gas filled microbubbles encapsulated with polymer, lipid or surfactant shells can be used as the most effective contrast agent for ultrasound imaging. Superparamagnetic iron oxide nanoparticles (SPIO) can be used as a powerful contrast agent for MRI. The combination of microbubbles and SPIOs, MMBs, can be used as the contrast agent for both US imaging and magnetic resonance imaging because the MMBs can overcome the shortcomings of magnetic nanoparticles or microbubbles, respectively. That is, the stability of microbubbles can be improved by embeding magnetic nanoparticles into the bubble shells. Moreover, the embedded nanoparticles can be delivered into desired regions under the guidance of magnetic field and can be released when suitable ultrasound exposure is chosen. Cai et al focused on the relationship between the MMBs structure and dual modality imaging, and gave a good overview on magnetic microbubbles for theranostics, including their preparation, imaging contrast agents (diagnostic) and drug delivery (therapeutic) [15].
Different from [15], the main contribution of this paper is the introduction of the above-mentioned dual modality contrast agent to multi-modality medical image registration. Using MMBs with the mean diameter of 3.98 μm prepared as described by Yang et al [2,6,16], this paper carries out the registration between US image and MRI, and comes to a conclusion that with the use of MMBs, the proposed algorithm performs well not only for global deformation but also for local deformation. The remainder of the paper is organized as follows: the registration method based on robust optical flow model between US image segmentation and MRI segmentation is described in Section 2. Section 3 provides several groups of comparison experiments, and analyzes the experimental results, while Section 4 concludes our paper.

Methods
As shown in Figure 1, MMBs were used in both ultrasound and magnetic resonance imaging process. In order to reduce the influence of the speckle noise on registration, semiautomatic segmentations of US image and MRI were carried out by active contour model. And then, a robust optical flow model between US image segmentation (floating image) and MRI segmentation (reference image) was built, and the vector flow field was estimated by the Coarse-to-fine Gaussian pyramid (see Figure 2) and graduated non-convexity (GNC) schemes. The registration method based on image intensity that directly uses gray information instead of the extraction feature process is widely concerned and quickly developed [17]. Image segmentation combined with optical flow algorithm can not only weaken the influence of noise, but also avoid feature extraction.

Image segmentation with active contour model
Because all US images are confounded by speckle noise, fully automatic segmentation of US image is currently impossible. Therefore, the focus has been on semi-automatic techniques, particularly active contour models which can detect objects whose boundaries are not necessarily defined by gradient. Active contour models are used to evolve a curve, subject to constraints from a given image under a number of external and internal forces [18][19][20][21][22]. The external forces attract the curve to regions of interest in the image, whereas the internal forces hold the curve smooth. If initialized close to a boundary, the curve deforms to "hug" the boundary along its length, providing further adaptability to noise.
Assuming the evolving curve is defined as C, it will move through the spatial domain of the given image I 0 to minimize the following energy function: here μ ≥ 0, ν ≥ 0, l 11 > 0, and λ 22 > 0 are fixed parameters, I 0 (x, y) is image intensity at pixel location (x, y), and the constants c 1 , c 2 (depending on C) are the averages of I 0 inside C and outside C, respectively. The length of the curve, Length (C), and the area of the region inside C, Area (inside (C)), are two regularizing terms. If the constant μ is larger, then only larger objects are detected, or objects that are grouped together. If it is small, then smaller objects will be detected. We do not want different objects close to each other to be interpreted as a single object. This is the reason we decided to set μ = 0 in Eq. (1). The minimization procedure uses iterative method and differential calculus. After initializing a curve close to the object boundary, the active contour model starts deforming to fit the local minimum so as to move towards the desired boundary and finally settles on it.

Robust optical flow model construction for registration
Let I 1 , I 2 denote the reference and floating images, namely MRI and US image. We need to determine the optical flow field (u, v) (the horizontal and vertical vectors) between I 1 and I 2 , which represents the displacement between the above two images. The classical optical flow objective function in its spatially discrete form is written as Where r D and r S are penalty functions, and l is a regularization parameter. Optical flow method is sensitive to noises because it is on the basis of differential technology, and some filters (both high-pass and low-pass) are used to reduce this bad effect [23]. After every iteration median filtering the intermediate optical flow field can effectively denoise the intermediate flow field, prevent gross outliers, and make non-robust methods more robust [24]. When the median filter is used to optimize the objective, it will lead to lower energies. Especially, the optimization of the classical model, with interleaved median filtering and an auxiliary flow field (û,v) [25,26], approximately minimizes where λ 1 is a regularization parameter, λ 2 and λ 3 are scalar weight, and N i,j is the set of neighbor pixels of pixel (i,j). By alternately minimizing (4) and (5), we can optimize the objective function as in (3): Hou et al. BioMedical Engineering OnLine 2015, 14(Suppl 1):S14 http://www.biomedical-engineering-online.com/content/14/S1/S14 By using the alternating optimization strategy, with (û,v) fixed, we minimize (4) with regard to (u, v); with (u, v) fixed, we minimize (5) with regard to (û,v) where . Coarse-to-fine Gaussian pyramid and graduated non-convexity (GNC) schemes are adopted to estimate (u, v) and (û,v) [24]. A two stage GNC process is adopted and 3 warping steps per pyramid level are performed. After every warping step, (u, v) are set to be (û,v). Finally, (û,v) are taken as the final flow vector field estimate. The framework of Gaussian pyramid algorithm is shown in Figure 2, and the penalty functions are set according to Sun et al [25].

Materials and data acquisition
Dual modality contrast agent MMBs were obtained from Jiangsu Laboratory for Biomaterials and Devices. MMBs can increase magnetic resonance susceptibility, and negatively enhance T2-weighted (T2*WI) imaging signal, namely, decrease the gray value of T2*WI imaging. They can also give strong ultrasound backscattering echo intensity and positively increase the brightness of US image.

Phantom
Phantom was made from glycerol, agar and water ratio of 3:4:90, and produced by Jiangsu Laboratory for Biomaterials and Devices, in which a "U" shaped silicone tube with external diameter of 9 mm and inner diameter of 7 mm is "vertically" sitting in the agar phantom. Three fatty objects which size is about a diameter of 5 mm were laid in the tube. When US imaging or magnetic resonance imaging with MMBs was carried on, a solution (0.1 g/ml) containing MMBs was injected into the silicone tube; while when US imaging or MRI without MMBs was carried on, purified water was injected into the silicone tube. Regardless of whatever MMBs is used, fluid in the tube remains stable during imaging.

Experimental computing platform
Our algorithm implementation is based on the compatible personal computer installed MATLAB 2008b. In some cases, C++ language compilation system was also needed to perform some functions, so it is necessary to install VISUAL C++ 6.0 (or VISUAL C+ + 2000.NET) on the above platform.
The three objects were imaged using the Ultrasonic imaging system of the GE LOGIQ3 PRO scanner (GE Medical System, USA) with a 4 MHz ultrasound transducer used as a transmitter as well as a receiver. B-mode US images were acquired with the instrument parameters (Gn 20; E/A 1/2; DR 78; AO 100%). T2*WI imaging of these objects was carried out with a clinical 0.3 T magnetic resonance imager (AIRIS II, Hitachi Ltd, JAPAN). Images were aquired with a matrix size of 256 × 256, field of view of 20 × 20 cm 2 , repetition time of 400 ms, section thickness of 4 cm, and echo time of 15 ms. The reasons the usage of 0.3 T magnetic resonance imager was as follows. Firstly, we had carried out multiple group comparison tests of magnetic resonance imaging using 1.0 T or above equipments based on MMBs, and obtained good imaging result. Secondly, now SIEMENS 0.3 T magnetic resonance imager was still used widely especially in underdeveloped area because of magnetic resonance imager's expensive upgrades. Figure 3 includes US and MRI images showing the effectiveness of MMBs. I, II and III in Figures 3(a)~(d) represent the above-mentioned three objects imaging. Without MMBs the tube boundary can't be seen, while with MMBs, the tube upper boundary and three targets can be seen from the B-mode image. Under ultrasonic conditions, gas imaging shows strong echo whereas liquid imaging has no echo, and MMBs almost float upward to the three targets and upper boundary of the tube, therefore the echo of the tube upper boundary and three targets is strong. The phantom surface is hard, and not easily deformed, and moreover the convex array probe is used, which lead to the poor contact between the probe and the phantom surface, and further bring lateral wall echo drop-out. Compared with Figure 3(a), the brighter objects of ROI can be seen distinctly in Figure 3(b). This illustrates qualitatively that MMBs can positively enhance the intensity of US. Similarly, the MRIs of the tube without and with MMBs were shown in Figure 3(c) & (d), respectively. The result indicates that MMBs can significantly reduce T2*WI signal intensity. Certainly, we can quantitatively measure objects brightness of US images and MRIs to determine how much MMBs influenced US imaging and magnetic resonance imaging [2].

Semi-segmentation of US images and MRIs
In Figure 4, when l 11 and l 22 are fixed to 1, and μ is fixed to 0, the segmentation results were compared with the change of parameter ν. The best segmentation was achieved when ν is set to 100 without MMBs. In contrast, we obtained the best segmentation when ν is set to 2000 with MMBs. image using MMBs than that not using MMBs, which is beneficial to segmentation of US image.
In Figure 6, we show how the active contour model works on the MRIs of phantom without and with MMBs, respectively. In our experiments, we fixed μ = 0, ν = 100 and λ 11 = λ 22 = 1. When MMBs are not used, 50 iterations and MRI segmentation are   Figure 6(d) has better contrast and smoother outline, showing MRI using MMBs is more beneficial to object segmentation. However, the difference between Figure 5(d) and Figure 5(d') is far more than that between Figure 6(b) and Figure 6(d), which shows the improvements of US image segmentation with MMBs is much better than the improvements of MRI segmentation with MMBs. MMBs have greater influence on US imaging than on magnetic resonance imaging.

US-MRI registration based on the proposed method
The registration results were compared by changing parameter λ 2 with/without MMBs when λ 1 and λ 3 are fixed to 5 and 1, respectively, as shown in Figures 7 & 8. According to Figures 7 & 8, the results of root mean square error (RMS), peak signal to noise ratio (PSNR), correlation coefficient (COR), and mutual information (MI) were quantitatively analyzed and given in Table 1.
When the reference and the floating image (or the registered image), are compared, the RMS of the pairwise differences of the two images can serve as a measure how far on average the error is from 0. When RMS is small, the similarity between the two images is greater.
Peak signal-to-noise ratio, often abbreviated PSNR, is an engineering term for the ratio between the maximum possible power of a signal and the power of corrupting noise that affects the fidelity of its representation. This ratio can be used as a quality measurement between the reference and the registered image. The higher the PSNR, the better the quality of the registered image.
COR is a mathematical measure of how much one image can expect to be influenced by changes in another. It is closely related to covariance. If there is no relationship between the two images the COR is very low.
MI is a fundamental concept in information theory, and a measurement about statistical correlation of two random variables. Consider gray values of two images which will be registered as two random variables, when the both images achieve the best registration, MI approaches the maximum. When RMS after registration is smaller than before registration, and three indices (PSNR, COR and MI) after registration is larger than before registration, it is called the normal variation, otherwise called abnormal variation in the following discussion. As shown in Table 1 we can achieve a better registration result independent of MMBs based on the proposed methods when λ 1 , λ 2 and λ 3 are set to 5, 1e+2 and 1, respectively. Figures 9 & 10 demonstrate US-MRI registration using the proposed method (λ 1 = 5, λ 2 = 1e + 2, λ 3 = 1) without and with MMBs, respectively. As it can be intuitively observed, Figure 10(f) has obvious advantages over Figure 9(f) from the perspective of registration. The four evaluation indexes (RMS, PSNR, COR and MI) in Table 2 had normal variations with and without MMBs, before and after registration, respectively. In addition, after registration, performance evaluation with the use of MMB had better improvement than that without MMB. For example, RMS decreases from 0.0843 to 0.0435, PSNR, COR and MI rises up from 21.4505 to 27.1892, from 0.3100 to 0.5303 and from 0.0629 to 0.0803, respectively. To sum up, the qualitative and quantitative analyses showed that US-MRI registration based on the proposed method is effective.  [27,28], particle swarm optimization (PSO) [29] and mutual information (MMI) [30,31]. From Table 2 without MMBs, before registration, RMS, PSNR, COR and MI are 0.1148, 18.7706, 0.2027 and 0.0437, respectively. After registration, for FFT and   Table 3 were improved, but not much. For MMI, the increase of RMS (from 0.1148 to 0.1156), the decrease of PSNR (from 18.7706 to 18.7069) and the decrease of COR (from 0.2027 to 0.1494) are all abnormal variation as shown in Table 3. From Figure 11, it can be seen that the registration results using  the other methods are unsatisfactory when MMBs are not used. The quantitative performance evaluated in Table 3 also confirms our intuitive feelings. In addition, with MMBs and before registration, Table 2 shows RMS, PSNR, COR and MI are 0.0657, 23.6063, 0.1415 and 0.0398, respectively. After registration, for FFT, PSO and MMI, the above evaluation indexes are improved.

PSO, the evaluation indexes used in
The above quantitative analyses also indicate that FFT is the best method among the other three methods, which is in accordance with the above intuitive observation as

Conclusions
In this study, MMBs were introduced as a new dual-modality contrast agent into the field of medical imaging. We verified that MMBs can increase the contrast of both US image and MRI, leading to the potential beneficial to registration of US and MR images. Using the same contrast agent for both US image and MRI would not only bring convenience to medical professionals, but also reduced health care cost. Qualitative and quantitative analyses of multiple group comparison experiments showed that registration results using all methods tested in this paper without MMBs were unsatisfactory. On the contrary, the proposed method combined with MMBs led to the best registration results. Our algorithm implementation was intensity-based and was independent of the metric used. Therefore, it can be adapted to different image modalities. At present, effort to improve algorithms for medical image processing has seen very little progress. Combining novel nanomaterials with algorithm optimization provides a new approach for potential gains in imaging processing.
Our results were encouraging. However, they were still at preliminary stage. Further in vivo studies including toxicological and pathological studies will be necessary before our methods could be implemented in clinical applications.