Overview of the proposed PRA workflow
The proposed augmented ultrasound-based image-guided framework comprises a diagnostic ultrasound device, an optical tracker with reflective markers, and a main computer. The proposed image-guided surgical workflow is as follows.
First, an MR scanning is performed on the patient, where the vessels exhibit high contrast relative to their background. The kidney, vessels, and skin are then extracted from the MR volume as a 3D model. Then surgeons can preoperatively define a needle trajectory that avoids vital vessels and facilitates an effective treatment. During the surgery, the US slices of the kidney are acquired at the maximum exhalation positions of each respiratory circle. The preoperative data and the surgical planning are then registered to the calibrated US images using two pairs of orthogonal slices, such that the planning can be transferred into the OR. The position of the needle tip is read by the optical tracker in real time. By augmenting the US with the MR models and a virtual needle, a 3D visual guidance is provided to facilitate the hand-eye coordination of the treating surgeon, such that the PRA can be performed accordance with the planning. The overview of the entire framework is shown in Figure
1. Next, we will describe in details the proposed framework.
The MR-based preoperative planning
In this subsection, we present a 3D visualized environment that combines convincing virtual representation of the kidney surface and the renal vascular structures from preoperative MR, allowing surgeons to plan an optimal needle trajectory. Here “optimal” means suitable and safe for the interventional puncture. The precondition of such a planning is to obtain abdominal volumetric MR images with high contrast and high spatial resolution and then to extract the 3D geometric description of the kidney and renal vessels. Thusly, we use the True Fast Image with Steady-state Precession (true-FISP) MR sequence
[12] to acquire the volume data, as shown in Figure
2a. Note that other contrast-enhanced imaging modalities such as MRA (Magnetic Resonance Angiography) are also suitable for the planning.
Due to the high contrast relative to the background, here we segment the vascular structures using a neighbourhood connected region growing algorithm for the sake of less manual interaction. Specifically, the algorithm starts with placing one or two seeds in the vessel region and incrementally segments the vessels by recruiting neighbouring voxels according to an intensity threshold
[13]. As the contrast-to-noise ratio of the kidney is not that high, the kidney is segmented in a manual way. After manually segmenting the skin, all segmented models are then smoothed using a 3D Gaussian kernel and converted into triangulated meshes by means of the marching cubes algorithm
[14]. Then, the triangulated meshes, including vessels, kidney, and skin, are fused and merged into one geometric model, as shown in Figure
2b. Based on such a visualized anatomy, a needle trajectory planning that is optimal for the PRA can be defined as an entry point on the skin and a target point in the kidney. Generally, the needle entry on the skin is near the 11th intercostal space, while the trajectory should avoid all large vessels.
The augmented US-based intraoperative guidance
We expect to provide intraoperative guidance by transferring the preoperative planning to the intraoperative conditions and augmenting the US with preoperative anatomical models. The critical problem here is to register the MR volume to the US slices considering the patient respiration. This problem can be formulated as follows:
Let x
US,i
be a pixel position in the intraoperative US plane while x
MR,j
be a voxel position within the preoperative MR volume. Given two point sets X
US = {x
US,i
} and X
MR = {x
MR,j
}, i = 1,…, N, j = 1,…, M, we aim to find a transformation T
R
that represents the spatial correspondence from the preoperative MR volume S
MR to the intraoperative space S
tra defined by the tracker. Based on T
R
, the preoperative models such as the planning and the 3D geometric anatomy can be transferred into the intraoperative space. Therefore, the puncture can be guided to be coincident with the planning. Next, we will describe the navigated intervention in detail.
(i). US calibration
The intraoperative processing starts with the US calibration that maps the pixel position set X
US to the intraoperative space S
tra. In particular, we first determine a homogeneous transformation T
C
that maps pixel positions from the 2D US slices to the local 3D space defined by the optically-tracked markers mounted on the US probe. Let y
US,i
denote the location of pixel x
US,i
in the local probe space, then we have
(1)
After a further transformation T
T
from the probe space to the tracker space, all US slices can be localized in the same space S
tra as
(2)
The calculation of T
C
and T
T
will be given later in Section 3. After the calibration, each pixel in all acquired US slices can be positioned in the intraoperative space as Z
US = {z
US,i
}.
(ii). Respiratory gating
Because of the organ motion due to respiration, inconsistency will occur between the positions of the anatomical features such as vascular structures in different US slices. This will lead to significant registration error. Therefore, we expect to acquire US slices at the same stages of the respiration cycles. It has been shown that for free respiration the kidney assumes the same positions at equivalent lung volumes
[15]. Moreover, the end-exhale represents the longest natural pause in a cycle
[16]. Thus, we expect to use only US slices at the maximum exhalation positions.
This can be achieved by an optical tracking based respiratory gating technique. First, the US probe is placed on the caudal end of the patient’s sternum, where the US slice plane is approximately parallel to the transverse section of the human body. At this position, we define a cranio-caudal line l
0 that is perpendicular to the transverse section of the human body and passes through the central point of the US transducer face. The line l
0 is considered as a reference axis. In order to acquire one US slice of interest at the maximum exhalation, the probe is placed at the specified location, and a certain number of slices are acquired and stored at a stable acquiring rate. Meanwhile, the instantaneous distance from the centroid of the optically-tracked markers mounted on the probe to the reference axis l
0 is calculated. The US slice with the minimum distance is considered as the required maximum exhalation slice, as shown in Figure
3. In such a case, the maximum exhalation serves as a trigger for slice selection, accounting for the breathing motion and ensuring consistency of all used US slices.
(iii). MR to US affine registration
We aim to transfer the preoperative planning onto the patient in situ by means of MR to US registration. With respect to clinical demand, surgeons prefer an effective image guidance that optimizes the surgeon's hand-eye coordination, but are often reluctant to accept over complicated workflow. To this end, we do not use a MR to densely-sampled US volume registration studied by several groups, which requires overlong intraoperative processing time
[17]. Meanwhile, we decide not to use the VNav method after careful consideration, which often suffer from less accuracy
[8]. Instead, we propose an efficient approach on top of an affine registration based on two pairs of orthogonal US images.
The affine registration is equivalent to find a transformation T
R
that best aligns the target point set X
MR with the source point set Z
US. The iterate closest point (ICP) algorithm
[18] is often used to estimate a transformation that minimizes the mean square distance between two point sets. However, the original ICP is sensitive to the initial pose and could be prone to fall into local minima, especially for noisy feature sets, e. g., a point-cloud selected in US images. Moreover, the free landmark extraction is often inefficient and leads to inaccurate registration, mainly because it utilizes insufficient “geometrical information”. In order to overcome these limitations, we propose an orthogonal-slice ICP (OICP) algorithm detailed as follows.
First, the US probe is swept near the 11th intercostal space. A few US images are acquired at the maximum exhalation positions based on the respiratory gating described above. Two pairs of almost orthogonal images, U
1, U
2, U
3 and U
4 are selected, where U
1, U
2 are approximately parallel to the transverse section of the body and U
3, U
4 are acquired with the probe along the midaxillary line. U
1, U
2 contain clearly visible hilum vein, inferior vena cava (IVC), and a transverse kidney contour while U
3, U
4 contain a clearly visible longitudinal kidney contour. Note that both the transverse and longitudinal contours should better be complete, or at least represent most of the kidney.
The kidney surface and large vessel surface are used as registration features for the best alignment of the US slices and the MR volume. The target features X
MR consist of kidney surface denoted as K
MR = {k
MR,j
}, j = 1,… K
1, and vessels including hilum vein surface and IVC surface denoted as V
MR = {v
MR,j
}, j = 1,… V
1. All are preoperatively segmented from the MR data, as shown in Figure
2b. The source feature Z
US is a set of manually-picked points, including a subset of point K
US = {k
US,i
}, i = 1,… K
2 selected on kidney contours (including transverse and longitudinal contours) in all used US slices, and a subset of points V
US = {v
US,i
}, i = 1,… V
2 on hilum vein and IVC surfaces in U
1, U
2. Because we only use two pairs of orthogonal slices, the point selection will be completed within reasonable time. The categorized feature data will aid the registration by avoiding local minima and reducing computation time. Let ck
i
and cv
i
be the closest points to k
US,i
and v
US,i
, respectively. Then, the OICP registration can be estimated as
(3)
The estimate starts with calculating an initiate transformation T
0 by aligning three pairs of landmark points selected from the cranial end, caudal end and kidney hilum on both the US slices and the MR segmentation images, as shown in Figure
4. The initial alignment can be calculated using the Procrustes analysis
[19]. This preprocessing allows quick algorithm convergence without falling into local minima. Assuming a fine initial registration, the following optimization can be constrained within a small translation range ± A and a small rotation angle range ± α, thus reducing the computing time while improving the reliability. The translation and rotation are then further optimized to minimize the mean square error (MSE). The entire optimization is summarized below.
Input: K
US and V
US manually picked from orthogonal US slices U
1, U
2, U
3 and U
4, K
MR, V
MR preoperatively segmented from the MR volume;
Output: T
R
;
Initiation: The starting registration T
0 is obtained by aligning the three pairs of landmark points.
Iteration: for n = 1 to n
max or until convergence do
-
1.
Compute the closest point for i = 1,…, K 2 and for i = 1,…, V 2;
-
2.
Compute an update T
n
that minimizes the MSE between T
n-1 Z US and {ck
i
}∪{cv
i
} with translation within ± A and rotation within ± α.
-
3.
End the iteration when n = n
max
or the decrease of the MSE is below a threshold h.
With the final estimate of T
R
, the transformation from the US plane to the MR image space S
MR can be given by
. Figure
5 shows the corresponding US and MR slices from a healthy volunteer, where the longitudinal kidney contour is well aligned.
(iv). Augmented US-based guidance
Based on T
R
, the planning and MR anatomical models can be transferred from the preoperative space S
MR into intraoperative space S
tra. A puncture trestle is mounted to the US probe to restrict the needle trajectory to several given angles within the US plane. The needle position can be precisely measured by the tracker in real time and fused in the guidance image as a virtual model. The visual guidance is provided by augmenting the US images with 3D anatomical models, the planning and the virtual needle, as shown in Figure
6.
One should notice that the transformation T
R
is calculated using only US slices at the maximum exhalation positions. Therefore, at the other stages of the respiratory circle, the accuracy of registered needle trajectory cannot be guaranteed because of the organ shift and soft-tissue deformation. In such a case, we expect to perform the puncture at maximum exhalation positions. To this end, we make use of the respiratory gating technique proposed above to plot the respiratory curve (as shown in Figure
3) in real time. The maximum exhalation positions can then be detected by visual inspection of the surgeon. Meanwhile, the needle is inserted rapidly into the intrarenal target. Under the presented image guidance, the puncture trajectory can be guided and guaranteed to be coincident with the planning.
Experiments and evaluations
The proposed image guidance framework is evaluated in two stages: (i) evaluation of the registration performance in terms of the accuracy, precision and processing time measured on human data, (ii) evaluation in terms of puncture accuracy and perceptual quality assessed by four urologists on kidney phantom trials.
(i). Registration evaluation
Here, we aim to measure the registration accuracy in terms of the root mean square (RMS) target registration error (TRE) on MR and US data provided by volunteers. Because no gold standard is available, the MR to densely-sampled US volume registration proposed in
[17] was used as a bronze standard
[20].
The true-FISP MR data was acquired by scanning four healthy volunteers (distinguished by A, B, C, and D) on a Siemens MAGNETOM Trio Tim 3.0 T machine. Written informed consent from all volunteers was obtained. The voxel resolution was set to 1.12 × 1.12 × 1.00 mm3 to approximate the isotropy. The US images were acquired using a Mindray DC-7 machine with a 3.5 MHz abdominal probe and then captured using an EDIUX NX video grabber from CANOPUS. A passive Polaris system from Northern Digital Incorporation (NDI) was used for position tracking. The US probes was mounted with optically-tracked reflective markers such that the position of the probe can be tracked. The Mindray US machine provides a built-in calibration application to output the distance from the left bottom of the US slice to the center of the arc-shaped probe surface, such that the transformation T
C
can be calculated directly. Note that for each imaging depth, the built-in US calibration algorithm only need to run once before the surgery. Then, the output can be stored in a mapping table between transformation T
C
and imaging depth for future use. The transformation T
T
was calculated automatically by means of the tracker’s real time output. Given T
C
and T
T
, all US slices can be calibrated and located in the tracker space S
tra, as described in section 2.
For acquiring one desired US slice at the maximum exhalation via the proposed respiratory gating, 50 to 100 slices were acquired at a stable rate of 20 slices per second using the video grabber. For calculating the bronze standard registration, 125 US images were selected at the maximum exhalation positions from each volunteer, covering from transverse to longitudinal views of the kidney. Four urologists with expertise in US-guided renal intervention were asked to individually conduct the proposed registrations and the bronze standard registration for each volunteer (denoted as Test 1–4). For calculating the OICP registration, the translation range and the rotation range are set to A = 28 mm and α = 30°, respectively. The threshold h for terminating the iterations was set to 0.1.
The accuracy can then be measured as follows. All voxel positions within the kidney model were transformed from S
MR into S
tra, using both the proposed registration method and the bronze standard. The RMS error between the two corresponding position sets was then calculated. Thus, the accuracy in terms of RMS TRE was obtained. To evaluate the precision, or repeatability of the proposed method, the RMS distance from the transformed positions to their average, i.e., the standard deviation, was calculated. The processing time for the proposed registration was also recorded, including the Procrustes initial registration and OICP optimization.
(ii). Phantom trials
A triple-modality (CT, MR, US) abdominal phantom model 057 from Computerized Imaging Reference Systems (CIRS) was used for the phantom trials. The internal structure of the model 057 includes partial abdominal aorta, partial vena cava, spine and two partial kidneys each with a lesion. The lesions are high contrast relative to the background in MR and can be barely identified in US. Each lesion can be punctured 3–5 times. These features make it a useful tool for evaluating the targeting accuracy of a multimodality image-guided PRA.
First, the phantom was scanned with the same Simons MR machine. The MR volume data was then pulled onto the augmented-US based guidance system for surgery planning. With the 3D reconstruction of the segmented kidney, lesions, spine and skin, two planning trajectories were defined, each including an entry point on the skin and a target point within a lesion.
The same four interventionists were asked to perform the RFA on the phantom (Test 1–4). Two pairs of orthogonal US slices containing longitudinal and transverse contours of the kidney were selected from real time US data for MR-US registration. Because the phantom is approximately rigid, the breathing gate technique was not used here. Three points from the cranial end, the middle, and the caudal end of the kidney were used for the initial alignment. 8 to 15 points from the kidney surface and the vena cava were used for the OICP registration. A rigid plastic needle equipped with four fixed optically-tracked markers was used. A puncture trestle was mounted to the US probe such that the needle trajectory was restricted, as shown in Figure
7. This needle can be located in real time in space S
tra and was intended to reach the target lesion. By observing the proposed visualized guidance, each interventionist performed two needle punctures. The environment of the phantom test is shown in Figure
8. After each PRA trial, MR scanning was performed to assess the puncture accuracy. The distance between the needle tip and the lesion center, denoted as needle-target distance (NTD), was measured based on the multiplanar reconstructed images.
The perceptual quality of the image guidance system for surgeons is very important, as their satisfaction relates to the therapeutic impact in selecting the system in clinical practice. The perceptual quality for the proposed PRA was rated in terms of three criteria according to a custom scoring system, as follows:
Intervention Improvement from 5 to 1 respectively denotes significant, meaningful, moderate, fair, and little localization improvement when the proposed guidance is employed. Workflow Impact indicates the acceptability of the proposed workflow, where 5 to 1 respectively denote positive, acceptable, acceptable after training, acceptable with reluctance, and unacceptable. Clinical Relevance denotes the clinical value of the proposed framework, where scores 5 to 1 correspond to values of high, moderately high, medium, moderately low to low. The evaluators were allowed to rate x.5 that represents an assessment between x and x + 1.