In this section, we first introduce the coronary IVUS dataset used for training and testing. Then, the 8-layer deep U-Net architecture that predicts the masks for the lumen and the EEM-CSA of IVUS images is presented. The training details are described, and the metric for evaluating the proposed method is illustrated.
Dataset and augmentation
We use the coronary IVUS dataset from The Second Affiliated Hospital of Zhejiang University School of Medicine. It consists of in vivo pullback of coronary artery acquired by the iLab IVUS from Boston Scientific Corporation equipped with the 40-MHz OptiCross catheter. It contains IVUS frames from 30 patients, which are chosen at the end-diastolic cardiac phase in DICOM formats, with the resolution of 512 × 512. The dataset is divided into two parts, 567 frames of 24 patients for training and 108 frames of 6 patients for testing, respectively. The training set is used for building the deep learning model and the testing set is used to evaluate the model performance.
IVUS images contain catheter, lumen, endothelium, intima, media, external elastic membrane, adventitia, atherosclerosis plaque. The external elastic membrane is usually treated as the borders of media and adventitia. The media is gray or dark as it contains dense smooth muscle. The adventitia is similar to external tissues surrounding the vascular walls. The endothelium and intima are thinner than the lumen and media. Thus, the lumen and EEM-CSA can be manually annotated by experienced physicians as the ground truth for metric evaluation. Each IVUS frame has been manually annotated for the lumen and EEM-CSA in the short-axis view by three clinical experts, daily working with the specific IVUS brand from the Cardiology Department, shown in Fig. 2. Each expert is blinded to the other two experts’ annotations and each frame is repeatedly labeled by each of the three experts to ensure the correctness and blindness of the annotations. From the visual point of view of annotation, 92% of the annotated cases have high consistency.
The training set comprises 567 frames, which is not large enough for training a CNN model from scratch. Data augmentation is essential for better performance. The augmentation is twofold and performed online. First, the coronary IVUS raw images and the corresponding ground truth are randomly (1) rotated at angles: 90°, 180° or 270°; (2) flipped up–down or left–right. Secondly, the MeshGrid is added to the raw image at pixel-level, providing the relative location information. Due to the relatively fixed position like intima and adventitia in IVUS images, MeshGrid could play a good guiding role in training process, which guides the neural network where to look.
Network architectures
The U-Net is one type of the fully convolutional network [13] and is the most common convolutional network architecture for biomedical image segmentation. It consists of encoder and decoder parts and predicts segmentation mask at pixel-level instead of image-level classification. The encoder part is used for down-sampling and extracts higher-level features. The decoder part is used for up-sampling the output from the encoder part and concatenates the feature maps of the corresponding layer by skip connection. The skip connection is to relieve the gradient diffusion problem due to deep layers. The final decoder layer is activated by softmax to produce the class probability map to recover the segment predictions.
The encoder part has 9 blocks and each incorporates two repeated operations of 3 × 3 convolution, batch normalization and LeakyReLU activation. The down-sampling operation of 3 × 3 convolution with stride 2 × 2 reduces feature maps by half. The size of the 8th block is 2 × 2 to capture the deeper abstract information. The decoder part has 8 blocks to restore the image dimension. Each up-sampling operation contains a 5 × 5 deconvolution with stride 2. The skip connection concatenates the corresponding feature maps. The last convolution outputs the probability map of mask class prediction by softmax activation. The entire architecture is shown in Fig. 3. The parameter initialization of all layers of the model uses the random initialization method.
Compared to other U-Net variations, our proposed U-Net was no major innovation in structure. We replaced the original 4-layer network with an 8-layer network, which been able to extract deeper image features. The actual results also confirmed this simple deepening design.
Implementation details
The model was trained and evaluated on Dell PowerEdge T640 server with Xeon Silver 4114 processor, 128 GB of RAM, and four Nvidia GTX 1080Ti graphics cards. It took less than 90 min for training and 10 ms per image for inference.
We implement model training with TensorFlow framework. The less frames are not enough to train the CNN from scratch, in addition to data augmentation, we also employed transfer learning to initialize the encoder of U-Net’s weights using VGG16[14]. The optimizer was Adam [15], which was fast and robust. The weights were initialized randomly and the batch size was set to 16. The initial learning rate was 0.001 with the decay of 0.1 every 2000 iterations. A total of 8000–10000 iterations were done for training. Lumen and EEM-CSA were trained and predicted at one shot with the softmax function as the output activation, which gave each pixel its class probability. The loss function was the sparse softmax cross entropy [16]:
$$L(p_{{\hat{y}}} ,p) = - \sum\limits_{j = 1}^{K} {p_{{\hat{y}_{j} }} \log (p{}_{j})} ,$$
(1)
$$p_{j} = {\text{softmax(}}x_{j} {)} = \frac{{e^{{x_{j} }} }}{{\sum\limits_{k = 1}^{K} {e^{{x_{k} }} } }},$$
(2)
with K being the number of classes, \(p_{j}\) being the predicted probability belonging to class j, and \(p_{{y_{j} }}\) being the true probability.
Evaluation criteria
In semantic segmentation, the mean intersection over union (MIoU) is a widely used metric to evaluate the model, which is a common measure in semantic segmentation [17]. We compute the MIoU score between the ground truth and the predicted masks:
$${\text{MIoU}} = \frac{1}{k + 1}\sum\limits_{i = 0}^{k} {\frac{{p_{ii} }}{{\sum\limits_{j = 0}^{k} {p_{ij} } + \sum\limits_{j = 0}^{k} {p_{ji} - p_{ii} } }}} ,$$
(3)
with k being the number of classes excluding background, and \(p_{ij}\) being the number of pixels of class i predicted to class j.