Reconstruction of freehand 3D ultrasound based on kernel regression
BioMedical Engineering OnLine volume 13, Article number: 124 (2014)
Freehand three-dimensional (3D) ultrasound has the advantages of flexibility for allowing clinicians to manipulate the ultrasound probe over the examined body surface with less constraint in comparison with other scanning protocols. Thus it is widely used in clinical diagnose and image-guided surgery. However, as the data scanning of freehand–style is subjective, the collected B-scan images are usually irregular and highly sparse. One of the key procedures in freehand ultrasound imaging system is the volume reconstruction, which plays an important role in improving the reconstructed image quality.
System and methods
A novel freehand 3D ultrasound volume reconstruction method based on kernel regression model is proposed in this paper. Our method consists of two steps: bin-filling and regression. Firstly, the bin-filling step is used to map each pixel in the sampled B-scan images to its corresponding voxel in the reconstructed volume data. Secondly, the regression step is used to make the nonparametric estimation for the whole volume data from the previous sampled sparse data. The kernel penalizes distance away from the current approximation center within a local neighborhood.
Experiments and results
To evaluate the quality and performance of our proposed kernel regression algorithm for freehand 3D ultrasound reconstruction, a phantom and an in-vivo liver organ of human subject are scanned with our freehand 3D ultrasound imaging system. Root mean square error (RMSE) is used for the quantitative evaluation. Both of the qualitative and quantitative experimental results demonstrate that our method can reconstruct image with less artifacts and higher quality.
The proposed kernel regression based reconstruction method is capable of constructing volume data with improved accuracy from irregularly sampled sparse data for freehand 3D ultrasound imaging system.
Image-guided technique has been widely used in minimally invasive surgery which can greatly reduce the patient’s pain, improve the operation success rate and accelerate postoperative recovery. Compared with computed tomography (CT) imaging and magnetic resonance imaging (MRI), ultrasound is more suitable in intraoperative imaging for its real-time imaging advantage. Further more, ultrasound is non-invasive, non-ionizing, portable, and low-cost.
As 2D ultrasound cannot provide complete volume data of tissues and organs, 3D ultrasound imaging system is put forward to overcome such limitations by constructing abundant 3D data. A number of approaches for constructing 3D ultrasound volume data have been proposed, which can be grouped into three categories: dedicated 3D probes, mechanical scanning and freehand scanning . 3D probes relies on an sophisticated mechanism for scanning a predefined region of interested (ROI), and can provide 3D volume data in real-time. Nevertheless they are expensive and have limitation on scanning large organs . The mechanical scanning is usually formed of a scanning head as well as a stepping motor, which move and rotate the 2D transducer and synchronously record positions and orientations data . Unfortunately, the scanning range is still limited . In comparison with the above approaches, freehand 3D ultrasound is relatively low-cost and flexible. The irregularly sampled B-scan images with corresponding positions and orientations are scanned and recorded by a 2D probe with transducer, which can be manipulated freely to view the desired anatomical section.
A number of algorithms for reconstructing 3D ultrasound images from these irregularly 2D B-scan images. These algorithms fall into three groups: Voxel-Based Methods (VBM), Pixel-Based Methods (PBM) and Function-Based Methods (FBM) . VBMs traverse all voxels in a target volume and inserts corresponding pixels from the input images. An usual method of implementation this way is the Voxel Nearest Neighbor (VNN) . VNN traverses each volume voxel and assigns the value of nearest image pixel. VNN method can preserve the most original texture patterns from B-scan images, but it also has a trend to generate large reconstruction artifacts when the distance of the voxel to the B-scan image plane is large. Another algorithm is the Distance-Weighted (DW) . DW traverses each volume voxel and assigns the value of average its local neighborhood pixels weighted by the inverse distances between those pixels and the voxel center. DW method trends to averaging, so it has the advantage of suppress speckle noise, however, it also smooth out the 3D image boundaries. In , Huang et al. introduce a median-filter-based reconstruction method to improve the quality of volume reconstruction by utilizing median filter to reduce speckle noise. In [2, 8–10], Huang et al. propose some algorithms by improving the DW algorithm. Such as the recently present a adaptive Gaussian distance weighted (AGDW) method to improve the quality of reconstructed image. The AGDW evaluates the homogeneity of the neighborhood for each voxel according to the local variance/mean of neighboring pixels. If a voxel is deemed as a homogeneous region, its neighboring pixels are averaged. Otherwise, the voxel value is calculated using an adaptive Gaussian distance weighted method with respect to the local statistics. However, the AGDW method trends to over-emphasize the effect of speckle reduction, the patch-like pattern can be observed from their experimental results. The Probe Trajectory (PT) method  builds on the DW algorithm. Instead of using the orthogonal projection of points to the nearest B-scans, a probe trajectory is estimated and used for finding the corresponding pixels in the nearest B-scans. PBMs traverse the input pixels and insert them into the corresponding target volume voxels. it consist of two steps: a Distribution Step (DS) and a Hole-Filling Step (HFS). An usual method of implementation this way is the Pixel Nearest Neighbor (PNN). In the DS, PNN traverses each pixel in all the 2D Ultrasound images, each pixel value is filled into the nearest voxel (Distribution Step). if multiple pixels contribute to the same voxel, the averaged value [12, 13], the maximum value , the most recent value  or the first value  are usually selected for assigned it. In the HFS, the volume from the DS has some empty voxels, each empty voxel is filled with the nearby already filled voxels. The usual method for this purpose is average or maximum [12, 16] or a median  of those voxels. Dewi et al.  propose a Hole-filling algorithm by improving the Olymplic operator. The algorithm shows a better empty voxel estimation than conventional Olympic, Average, and Maximum. The PNN method causes great blurring as well as relatively large reconstruction error. In , a method named direct frame interpolation (DFI) method is presented to creates additional intermediate image frames by directly interpolating between two or more adjacent image frames of the original image series. Then the DFI method fills the the target volume using the original frames in combination with the additionally constructed frames. The DFI method requires the sampled B-scan slices not to be intersected. However, the intersection among the B-scan slices could not be avoided in the freehand scanning configuration. In , a method based on Cyclic Regularized Savitzky-Golay (CRSG) filter is introduced to reconstruct freehand 3D ultrasound volume with mechanical linear scanning sweep. However, such parallel scanning sweep is usually not available for the freehand scanning. To preserve the detail of boundaries, An Fast Marching Method (FMM) algorithm has been recently proposed by , it consists of DS and HFS. In DS, it similar to DS of PNN, In HFS, it advances the interpolation boundary along its normal direction and fills the area closest to known voxel points in first. it improvement in reconstruction accuracy and efficiency. FBMs utilize estimate functions by taking use of the input data for creating the voxel grid. Rohling et al.  propose the Radial Basis Function (RBF) interpolation is an approximation with splines that tries to use the underlying shape of the data in the volume reconstruction. Overfitting phenomenon is usually unavoidable for the Spline method. Sanches and Marques  present the Rayleigh reconstruction/interpolation with a Bayesian framework estimates a function for the tissue by statistical methods. The Rayleigh method also trend to suppress spckle noise but smooth out the boundaries.
This paper develops a new freehand 3D ultrasound reconstruction algorithm based on kernel regression (KR) aims to suppress speckle noise and preserve boundaries. its corresponding ideas come from nonparametric estimation . Nonparametric estimation has various applications, such as 2D image denoising, upscaling, interpolation and fusion , feature extraction for 3D MR image denoising . In the following, we demonstrate the details of KR applied in freehand 3D ultrasound reconstruction.
System and methods
The freehand 3D ultrasound system consists of three modules: a conventional 2D ultrasound scanner (DC-7, Mindray Medical International Ltd., Shenzhen, China) used to acquire ultrasound image, an electromagnetic spatial sensing device (Aurora , NDI, Ontario, Cannada) acquiring the position and orientation of ultrasound images and surgical instruments, and a workstation with custom-designed software used for data collection, volume reconstruction, and visualization . Figure 1 illustrate this system framework. The spatial information (position (x,y,z) and orientation (R x ,R y ,R z )) of ultrasonic probe and surgical instrument embeding sensor is recorded by Aurora system connects to the workstation through its USB port so that it can be acquired by a custom-designed software by use of Aurora system API. Besides, the real-time ultrasound video is captured by a video capture card (RGB-133, VTimage Inc., Shenzhen, China) installed in the workstation.
Before volume reconstruction, data collection is a very important step which influences accuracy and efficiency of reconstruction. Data collection consists of two steps: calibration between spatial data and 2D B-scan images, and selecting ROI (region of interest). In this paper, the set of B-scan image I i and its position T i are collected by the data collection method of .
A 3D volume data is reconstructed from the collected data, including the 2D B-scan images and its spatial information. In this study, the algorithm for volume reconstruction is composed of two stages: bin-filling and regression.
The bin-filling stage is to map the pixel in 2D B-scans into the voxel in 3D volume data based on its corresponding positional information. In this freehand 3D ultrasound system, the mapping of the coordinate system from the 2D B-scans to 3D volume is named after forward mapping and is defined as
where V p is the physical position, M is the forward transformation matrix, and V r is the resulting voxel location in the reconstructed 3D volume data. The forward transformation matrix M must be decomposed and implemented to find an matrix for this transformation, which is discussed in detail in .
Since the scanning of freehand-style is subjective, the collected B-scan images are usually irregular and highly sparse. Therefore, there are some gaps in the reconstructed volume after the bin-filling stage, as addressed in Figure 2(a). The goal of the regression stage is to make the nonparametric estimation  for the whole volume data from the previous sparse volume data.
The sparse volume data after the bin-filling stage are given by
where r(·) is the regression function, X i = (X i 1,X i 2,X i 3)T is the three-dimension coordinate of data, ε i s are the independent and identically distributed zero mean noise values.
Specifically, if X is near the sample at X i , we can approximate it with a N-term Taylor series
A least-squares formulation capturing this idea is to solve the folowing optimization problem:
where K(·) is the kernel function which penalizes distance away from the local position where the approximation is centered, and the smoothing parameter h (bandwidth) controls the strength of this penalty. In particular, the function K is a symmetric function which attains its maximum at zero, satisfying
where c is a constant value. The choice of the particular form of the function K is usually Gaussian, exponential, or other forms, which comply with the above constraints. Because the choice of the kernel has little impact on the accuracy of estimation. Therefore, the Gaussian kernel, being computational complexity, is selected in this paper.
For the estimation problem based on Least Square Method upon showed in Equation 4, the order N effect the accuaracy and complexity of local approximation of the volume data. Therefore, it must be appropriately chosen. In the nonparametric statistics literature, locally constant, linear, and quadratic approximations (corresponding to N=0,1,2) have been considered most widely [24–27].
The Kernel function K is now a function of 3 variables. Given a nonsingular positive definite 3 × 3 bandwidth matrix H, which is defined
Often, one scales each covariate to have the same mean and variance, then use the kernel
where K is any one-dimensional kernel. therefore, there is a single bandwidth parameter h. At a target value X = (X 1,X 2,X 3)T, the local sum of squares is given by
The estimator is
where is the value of β = (β 0,⋯,β 3)T that minimizes the weighted sums of squares. The solution is
and W x is the diagonal matrix whoses (i,i) element is w i (X). Therefore,
where e 1 is a column vector with all elements equal to zero but the first element equal to one, is the final values of reconstructed volume data.
Experiments and results
To evaluate the quality and performance of our proposed KR algorithm for freehand 3D ultrasound reconstruction, a phantom and an in-vivo liver organ of human subject are scanned with our freehand 3D reconstruction system. The parameters of the order N, the size of kernel R and the bandwidth h of KR are assigned with 1, 15 and 0.5 respectively. The proposed method is compared with the classical VNN, PNN, Spline and PT methods and the recent FMM method. The accuracy of the reconstruction results is evaluated via root mean square error (RMSE). The running time of different methods on all scanned datasets are given for their performance and efficiency comparison. All of the methods are implemented in C++, and the computer is equipped with a Intel Core i3-2120 3.3 GHz and 2 GB RAM.
The first experiment is conducted on an abdominal phantom (CIRS Model 057), which is made from proprietary materials to accurately mimic human tissues under MRI, CT, and ultrasound. The phantom mainly contains simulated lungs, liver, hepatic vesels, ribs, vertebra, kidneys, and abdominal aorta. The freehand scan of the phantom is performed with a 4.5 MHz probe. 102 B-scan slices are collected. Each B-scan is cropped to 400 × 400 pixels. A depth setting of 200 × 200 mm is used giving a resolution of 0.5 mm/pixel. Figure 3 shows the phantom picture, its inner structure and the reconstructed volume data with our freehand 3D ultrasound reconstruction system.The second experiment object is an in-vivo liver organ of human, 167 B-scan slices are collected. each B-scan is cropped to 347 × 242 pixels. A depth setting of 177 × 123 mm is used giving a resolution of 0.511945 mm/pixel. Figure 2 shows the bin-filled volume without interpolation and the reconstructed volume using our KR algorithm.
The coronal, sagittal and transverse slice of the phantom and liver using the VNN, PNN, Spline, PT, FMM and KR algorithm are showed in Figure 4 and Figure 5. According to Figure 4(b) and Figure 5(b), the reconstructed slices using the VNN algorithm look sharp and present more texture patterns. However, the anatomical structure is actually distorted due to the misalignments of the pixels, as addressed by . Figure 4(c) and Figure 5(c) show the PNN method causes great blurring as well as relatively large reconstruction error. Overfitting phenomenon is usually unavoidable for the Spline method, as demonstrated in Figure 4(d) and Figure 5(d). Figure 4(e) and Figure 5(e) show the PT method preserve more texture details. However, there are some gaps in the sagittal slice. Figure 4(f) and Figure 5(f) show the recent FMM method performs much better than the classical methods. However, it can’t preserve enough details in the big gap. Compared with the classical methods and the recent FMM method, our KR algorithm has advantages both in recovering missing data with more details and suppressing speckle noises, as demonstrated in Figure 4(g) and Figure 5(g).
For the quantitative evaluation, we deliberatly remove a slice from the collected B-Scan slices. Then different reconstruction algorithms are applied to the removed data. Such evaluation method is capable of measuring the ability of a reconstruction algorithm in preserving true intensity values at the locations where a part of original data is removed. A good reconstruction algorithm should interpolate the removed data with values very close to the original data. In this study, root mean square error (RMSE)  is used for the image quality measure. It can be described as follow:
where I o is the removed slice from B-scan images, I r is an image resliced the reconstructed volume with I o ’ position. (x,y) is the coordinate of pixels.
Table 1 and Table 2 summarize the RMSE of the evaluation tests using the VNN, PNN, Spline, PT, FMM and our KR reconstruction algorithms. The RMSE of our KR algorithm is smallest among these reconstruction mehods. Figure 6 and Figure 7 show that the KR based reconsturcted slice is more close to the original slice.
Computation complexity analysis
In order to perform an objective and theoretical estimations to the computation time, the big O notation is used to identify the complexity function. Table 3 lists the computational time complexity of VNN, PNN, Spline, PT, FMM and our KR approaches.
The practical time costs are demonstrated in Table 4 for the phantom test (with a dimension of 424×421×131) and Table 5 for the liver test (with a dimension of 387×338×350). From Table 4 and Table 5, the KR takes more computation time than VNN, PNN, Spline, PT, and FMM algorithms.
In this paper, A novel freehand 3D ultrasound system based on kernel regression is proposed. A phantom and an in-vivo liver organ of human subject are scanned and reconstructed with this system for experiments. Root mean square error (RMSE) is used for the quantitative evaluation. Both of the qualitative and quantitative experimental results have demonstrated that our method can reconstruct image with less artifacts and higher quality than the classical VNN, PNN, Spline, PT methods and the recent FMM method. Therefore, the proposed kernel regression based reconstruction method is capable of constructing volume data with improved accuracy from irregularly sampled sparse data for freehand 3D ultrasound imaging system. Of course, the practical time cost experiments show it costs more computation time. So the time complexity is required to improve for higher efficiency in freehand 3D ultrasound reconstruction in the future work. More over, in respect to quality, automatic and dynamic bandwidth will be studied for spatial adaptation.
Voxel nearest neighbor
Pixel nearest neighbor
Fast marching method
Radial basis function
Region of interest.
Wen T, Zhu Q, Qin W, Li L, Yang F, Xie Y, Gu J: An accurate and effective fmm-based approach for freehand 3d ultrasound reconstruction. Biomed Signal Process Control 2013,8(6):645–656. 10.1016/j.bspc.2013.05.009
Huang Q-H, Zheng Y-P, Lu M-H, Chi Z: Development of a portable 3d ultrasound imaging system for musculoskeletal tissues. Ultrasonics 2005,43(3):153–163. 10.1016/j.ultras.2004.05.003
Toonkum P, Suwanwela NC, Chinrungrueng C: Reconstruction of 3d ultrasound images based on cyclic regularized savitzky–golay filters. Ultrasonics 2011,51(2):136–147. 10.1016/j.ultras.2010.07.003
Solberg OV, Lindseth F, Torp H, Blake RE, Nagelhus Hernes TA: Freehand 3d ultrasound reconstruction algorithms-a review. Ultrasound Med Biol 2007,33(7):991–1009. 10.1016/j.ultrasmedbio.2007.02.015
Sherebrin S, Fenster A, Rankin RN, Spence D: Freehand three-dimensional ultrasound: implementation and applications. In Medical Imaging 1996. Bellingham: International Society for Optics and Photonics; 1996:296–303.
Rohling R, Gee A, Berman L: A comparison of freehand three-dimensional ultrasound reconstruction techniques. Med Image Anal 1999,3(4):339–359. 10.1016/S1361-8415(99)80028-0
Huang Q-H, Zheng Y-P: Volume reconstruction of freehand three-dimensional ultrasound using median filters. Ultrasonics 2008,48(3):182–192. 10.1016/j.ultras.2007.11.005
Huang Q-H, Zheng Y-P: An adaptive squared-distance-weighted interpolation for volume reconstruction in 3d freehand ultrasound. Ultrasonics 2006, 44: 73–77. 10.1016/j.ultras.2005.08.004
Huang Q, Lu M, Zheng Y, Chi Z: Speckle suppression and contrast enhancement in reconstruction of freehand 3d ultrasound images using an adaptive distance-weighted method. Appli Acoustics 2009,70(1):21–30. 10.1016/j.apacoust.2008.02.002
Huang Q, Zheng Y, Lu M, Wang T, Chen S: A new adaptive interpolation algorithm for 3d ultrasound imaging with speckle reduction and edge preservation. Comput Med Imaging Graph 2009,33(2):100–110. 10.1016/j.compmedimag.2008.10.006
Coupé P, Hellier P, Morandi X, Barillot C: Probe trajectory interpolation for 3d reconstruction of freehand ultrasound. Med Image Anal 2007,11(6):604–615. 10.1016/j.media.2007.05.002
Nelson TR, Pretorius DH: Interactive acquisition, analysis, and visualization of sonographic volume data. Int J Imaging Syst Technol 1997,8(1):26–37. 10.1002/(SICI)1098-1098(1997)8:1<26::AID-IMA4>3.0.CO;2-V
Gobbi D, Peters T: Interactive intra-operative 3d ultrasound reconstruction and visualization. Med Image Comput Computer-Assisted Intervention-MICCAI 2002, 2489: 156–163.
Ohbuchi R, Chen D, Fuchs H: Incremental volume reconstruction and rendering for 3-d ultrasound imaging. In Visualization in Biomedical Computing. Bellingham: International Society for Optics and Photonics; 1992:312–323.
Trobaugh JW, Trobaugh DJ, Richard WD: Three-dimensional imaging with stereotactic ultrasonography. Comput Med Imaging Graph 1994,18(5):315–323. 10.1016/0895-6111(94)90002-7
Estépar R, Martín-Fernández M, Alberola-López C, Ellsmere J, Kikinis R, Westin C-F: Freehand ultrasound reconstruction based on roi prior modeling and normalized convolution. Medic Image Comput Computer-Assisted Intervention-MICCAI 2003,6(Pt 2):382–390.
Dewi D, Wilkinson M, Mengko T, Purnama I, van Ooijen P, Veldhuizen A, Maurits N, Verkerke G: 3d ultrasound reconstruction of spinal images using an improved olympic hole-filling method. In Instrumentation, Communications, Information Technology, and Biomedical Engineering (ICICI-BME), 2009 International Conference On. Bandung: IEEE; 2009:1–5.
Scheipers U, Koptenko S, Remlinger R, Falco T, Lachaine M: 3-d ultrasound volume reconstruction using the direct frame interpolation method. Ultrason Ferroelectr Freq Control IEEE Trans 2010,57(11):2460–2470.
Sanches JM, Marques JS: A rayleigh reconstruction/interpolation algorithm for 3d ultrasound. Pattern Recognit Lett 2000,21(10):917–926. 10.1016/S0167-8655(00)00053-2
Fan J: Local linear regression smoothers and their minimax efficiencies. Annals Stat 1993,21(1):196–216. 10.1214/aos/1176349022
Takeda H, Farsiu S, Milanfar P: Kernel regression for image processing and reconstruction. Image Process IEEE Trans 2007,16(2):349–366.
López-Rubio E, Florentín-Núñez MN: Kernel regression based feature extraction for 3d mr image denoising. Med Image Anal 2011,15(4):498–513. 10.1016/j.media.2011.02.006
Triple Modality 3D Abdominal Phantom Model 057 [ http://www.cirsinc.com/products/all/65/triple-modality-3d-abdominal-phantom/]
Hardle W: Applied Nonparametric Regression, vol. 5. Cambridge England and New York: Cambridge University Press; 1990.
Härdle W: Smoothing Techniques: with Implementation in S. New York: Springer; 1991.
Wand MMP, Jones MC: Kernel Smoothing, vol. 60. UK: Crc Press; 1995.
Hardle W: Nonparametric and semiparametric models. Springer, Verlag Berlin and Heidelberg GmbH & Co. K; 2012.
Wang Z, Bovik AC: A universal image quality index. Signal Process Lett IEEE 2002,9(3):81–84.
This study was financed partially by the National Natural Science Foundation of China (NSFC: 61401451, 61472411, 61103165, 81171402), the National 863 Program of China (Grant No. 2012AA02A604), the Next generation communication technology Major project of National S&T (Grant No. 2013ZX03005013), the Key Research Program of the Chinese Academy of Sciences, and the Guangdong Innovation Research Team Funds for Image-Guided Therapy and Low-cost Healthcare.
Chen is interested in computer graphics and image processing. His research focuses on medical image processing.
XKC carried out the algorithm design and implementation, and drafted the manuscript. TXW participated in the design of the study and the system framework. WJQ, DL, WZP, XL and JG conceived of the study, and contributed to discussions and suggestions to complete the manuscript. All authors read and approved the final manuscript.
Xiankang Chen, Tiexiang Wen contributed equally to this work.
Authors’ original submitted files for images
Below are the links to the authors’ original submitted files for images.
About this article
Cite this article
Chen, X., Wen, T., Li, X. et al. Reconstruction of freehand 3D ultrasound based on kernel regression. BioMed Eng OnLine 13, 124 (2014). https://doi.org/10.1186/1475-925X-13-124