Data sources
We used the Lung Image Database Consortium and Image Database Resource Initiative (LIDC-IDRI) database as the source of pulmonary nodule data. This database has the largest number of public lung images, and contains complete lung CT image slices and the specific annotation information of all nodules in image slices from 1007 patients. The LIDC-IDRI database was collected and published by the American National Cancer Institute to serve as an international research resource to aid research of early lung cancer [12]. Each patient has an eXtensible Markup Language (XML) format file. These files contain detailed information regarding the number of pulmonary nodules, their location, and characteristics, as interpreted by four radiologists. The characteristics deemed appropriate for diagnosis of pulmonary nodules include subtlety, internal structure, calcification, sphericity, margin, lobulation, spiculation, texture, and malignancy [13]. LIDC–IDRI includes all types of pulmonary nodules, such as solid nodules, part-solid nodules, and ground glass nodules. This database was used by the computer for the deep learning process to identify nodules.
For our study, all the GGO images in the database were extracted based on the characteristics of internal structure and texture [13]. Besides, Xinhua Hospital also provided 221 GGO images of 154 patients from 2016 to 2017 to expand the sample size. The cases of GGOs provided by Xinhua Hospital were identified and confirmed by two radiologists and two respiratory physicians, to ensure accuracy.
Pulmonary region extraction
On the CT image, the pulmonary parenchyma includes the bronchus and bronchoalveolar structures. We analyzed CT slices to identify pulmonary nodules; therefore, we only focused on the lung parenchyma, and not on the external contour. In order to minimize the error of the external contour on the experimental results, the lung parenchyma was extracted by threshold binarization, extraction of the maximum connected component, and separation of the adhesions between the pulmonary nodules and pleura and the pulmonary contour by means of corrosion (Fig. 1). Only the lung parenchyma was retained for subsequent analysis.
Nodule extraction
After analysis and extraction of the pulmonary parenchyma, we could determine the position of the candidate nodule. Taking the centroid location as the center, we cut out 64 * 64 small blocks from the lung parenchyma, which were regarded as the regions of interest (ROIs) (Fig. 2).
ROI superposition
Large nodules were easy to find; however, some smaller nodules and GGOs were similar to normal lung tissue on the image. In our study, in order to better differentiate between the pulmonary nodules (especially smaller nodules) and normal lung tissue, we used three continuous CT slices to attain more features of the ROI. After the centroid of a candidate pulmonary nodule was obtained, upper and lower CT slices were extracted from the initial CT slice, the three ROI pieces were superimposed on red, green, and blue channels (RGB), respectively, and pseudo color images were formed. Because of the spheroidal characteristics of pulmonary nodules, the three consecutive slices could be approximately overlapped, and the superposed RGB images were also spherical. Normal tissues, such as blood vessels, were seen as longitudinal stripes, and most of them were not perpendicular to the horizontal surface; therefore, they had a distinct RGB change after superposition (Fig. 3). After RGB channel superposition of the ROI, we could see the longitudinal trend of some tissues on 2D image. This method significantly enhanced the differentiation between pulmonary nodules and normal tissue.
Deep learning
In this study, we used AlexNet [14] and GoogLeNet [15] to detect pulmonary nodules, and ResNet50 [16] to detect GGOs. Convolutional architecture for fast feature embedding (CAFFE) [17] was used as the framework for deep learning; it was developed by Berkeley Vision and Learning Center. CAFFE has the advantages of fast operation and high extensibility. The operating system was CentOS 7.3 and the GPU video card used was GeForce GTX 1080 N (NVIDIA, Santa Clara, CA).
Deep learning of pulmonary nodules
We used the LIDC–IDRI database as a sample of CT slices for deep learning training. We used more than 10,000 ROI pseudo color images of pulmonary nodules extracted from the CT slices of 800 patients and about 12,000 ROI pseudo color pieces of normal pulmonary tissue as a training sample set. After we achieved a prediction model through deep learning training, another 176 patients’ CT images from the training sample set were used as the testing sample set. There were 321 pulmonary nodules in the testing sample set.
Deep learning of GGOs
We extracted 1293 ROI pictures of GGOs from the LIDC–IDRI database based on nodule characteristics in the XML files and confirmation by two radiologists and two respiratory physicians. Of the 1293 samples of GGOs, 1000 ROI pictures were placed in the training set and 293 in the testing set. Because of the small sample size of GGOs in the LIDC–IDRI, Xinhua Hospital also provided another 221 pictures of GGOs from 154 patients to expand the training set. Finally, there were 1221 pictures of GGOs and 1200 of non-GGOs in the training set, and 293 pictures of GGOs and 300 of non-GGOs in the testing set.
The specific steps involved are shown in Fig. 4.
Prediction models
AlexNet [14] and GoogLeNet [15] were used to detect pulmonary nodules. The ResNet and the pre-trained ResNet models were used as prediction models for GGOs.