Deep learning and level set approach for liver and tumor segmentation from CT scans

Abstract Purpose Segmentation of liver organ and tumors from computed tomography (CT) scans is an important task for hepatic surgical planning. Manual segmentation of liver and tumors is tedious, time‐consuming, and biased to the clinician experience. Therefore, automatic segmentation of liver and tumors is highly desirable. It would improve the surgical planning treatments and follow‐up assessment. Method This work presented the development of an automatic method for liver and tumor segmentation from CT scans. The proposed method was based on fully convolutional neural (FCN) network with region‐based level set function. The framework starts to segment the liver organ from CT scan, which is followed by a step to segment tumors inside the liver envelope. The fully convolutional network is trained to predict the coarse liver/tumor segmentation, while the localized region‐based level aims to refine the predicted segmentation to find the correct final segmentation. Results The effectiveness of the proposed method is validated against two publically available datasets, LiTS and IRCAD datasets. Dice scores for liver and tumor segmentation in IRCAD datasets are 95.2% and 76.1%, respectively, while for LiTS dataset are 95.6% and 70%, respectively. Conclusion The proposed method succeeded to segment liver and tumors in heterogeneous CT scans from different scanners, as in IRCAD dataset, which proved its ability for generalization and be promising tool for automatic analysis of liver and its tumors in clinical routine.

about the abdominal organs in the human body. 2 Liver organ and tumor segmentation from CT scans is an important step in the visualization of liver anatomy for surgical planning. 3 Manual segmentation of liver organ and tumors from CT scans is tedious and time-consuming. It greatly depends on the skills of the physician or doctor who performs the segmentation task. Liver organ has high variability in terms of shape and volume between different patients. 4 Low contrast and blurry edges are the main characteristics of CT images, which make liver delineation a challenging task. 5 Tumor segmentation adds more challenge due to the small observable changes between tumor and healthy tissues especially at their borders. In addition, tumors vary greatly in terms of shape, size, and texture. Despite these challenges, which complicate tumor segmentation, the automated approach is desirable, as it is, ideally, more objective and removes dependence on human skill. 6 Liver surgical planning treatments would benefit from an accurate and fast liver and tumor segmentation that allows for subsequent determination of tumor burden and texture-based information.
Moreover, having a standardized and automatic segmentation method would facilitate a more reliable therapy response classification. 7 Organ segmentation from CT scans has been a hot research topic during the past few years. Recently, due to the advancement in computer vision, the development of deep fully convolutional neural (FCNs) networks enhanced the performance of the semantic segmentation, and leads to outperform other competitors in the field of medical imaging. 8,9 General FCN focuses its task on image classification, where input is an image and output is one label. However, in medical imaging, it requires, besides the classification, to localize the area of abnormality. 10 Following the FCN success, many attempts have been carried to use the FCN for liver and tumor segmentation 11 ; one of the best FCN architectures has been created is the U-Net. 12 U-Net is succeeded to classify the images and locate the specific structures, and it has the ability to locate and distinguish borders by doing classification on every pixel.
Recently, the liver tumor segmentation (LiTS) competition challenge was organized in conjunction with ISBI 2017 conference. 13 The top-rated automatic methods submitted to the competition used FCN networks. For this purpose, different works used U-Net architecture for liver and tumor segmentation. 11,14,15 Despite the high accuracy achieved by deep learning FCNs in segmenting organs from CT scans, these methods depend on the training step on many datasets to cover all expected features of the intended organ and build a trained network to detect that organ in the test dataset. However, these methods overlook to get benefit of the local features in the test dataset itself to refine and improve the final segmentation from the target CT scan.
Due to the previously explained issue, the recent research moves toward the combination between deep learning methods with local information-based techniques. In liver and tumor segmentation domain, many intensity-based techniques have been proposed to find the intensity range of the liver and tumor by applying a statistical analysis on the intensities in CT scans. 16,17 One of these techniques that can be used more independently is the level setbased active contour methods. 18 Level set-based active contour method is used to deform an initial mask, that is coarse segmentation, to match more accurately the boundary of the liver/tumor in the test CT scan. 19,20 In this work, two cascaded FCN networks are constructed using U-Net, the same work proposed by Christ et al. 21 The first subnetwork aims to locate and predict the liver organ, while the second subnetwork work on the segmented liver envelope to detect and segment tumors. The output of these networks represents the coarse segmentation of liver and tumors, that are considered as initial mask used by the level set method to be deformed to the liver/ tumor boundaries in the target CT image and generate the final segmentation.

2.A | Overview of the proposed framework
The proposed segmentation framework is presented in Fig. 1. The workflow consists of three main steps. It is applied first for liver segmentation and then for tumor segmentation. The first step (Section 2.B) deals with data preprocessing, windowing, and filtering steps are applied on LiTS datasets for liver and tumor segmentation.
In a second step (Section 2.C), U-Net FCN is constructed and trained-one network to segment liver organ and another network is trained to segment the tumors inside the liver region of interest (ROI). In the third step (Section 2.D), the localized level set is applied on the predicted U-Net segmentation for further enhancement to get the final liver and tumor segmentation.

2.B | Data preparation
For liver segmentation step, the contrast-enhanced CT scans undergo median filtering to improve intensity homogeneity especially in liver region.
Step of intensity windowing is applied to exclude irrelevant organs and focus on liver organ intensity range. Based on intensitybased techniques, [16][17][18][19][20] liver organ and tumors Hounsfield (HU) intensities range is 0-200. In this work, HU windowing is applied on datasets used in the FCN training step for both liver and tumor segmentations, the used HU window is −50-250. Figure 2 presents the effect of applying windowing and median filtering on CT slice example.
For tumor detection and segmentation using U-Net network, the training datasets are enhanced using tensor-based 3D edge enhancing diffusion (EED) filtering, 22 that would improve the prediction of U-Net network to detect and segment tumors. EED filtering uses diffusion tensor to adapt the diffusion based on the image structure.
Edge enhancing diffusion filter is used to enhance the contrast, filters the noise in the homogeneous regions, and preserves the boundaries of the shape. 22 Edge enhancing diffusion filtering enhances the contrast of tumors by enhancing the homogeneities inside the liver and tumor tissue regions. In addition, it preserves the boundaries between tumors and liver tissue. Figure 3 shows an example of a CT scan before and after being enhanced using EED filtering. The intensity of the liver parenchyma is enhanced and appears brighter than the tumor regions, while the tumors appear darker compared to the liver tissue.
In this work, EED filtering improves the contrast of tumor structures by enhancing the homogeneities inside the liver and tumor tissue regions, and preserves the boundaries between them. This step would teach and orient the U-Net FCN network to extract and learn the features that differentiate the tumor structure from the surrounding tissues. Figure 3 shows the effect of the preparation step on the raw medical CT slice.

2.C | Fully convolutional neural networks
In this work, the U-Net architecture is used to build the FCNs. The networks are used to compute the soft probability label maps. Both U-Nets enable accurate pixel-wise prediction by combining spatial and contextual information in a network architecture comprising 19 convolutional layers. Figure 4 shows the U-Net architecture, the input passes and is processed by a sequence of convolution blocks, where the feature maps are doubled and resolutions are decreased (contracting path).
The expanding path of the U-Net reverses the process using the transposed convolution. The network contains dropout layer (0.5) before the final output layer to avoid over fitting. The output layer is designed using a linear classifier, sigmoid, that outputs a probability value (0-1) for each pixel being liver (tumor) or the background. The U-Net FCN architecture is implemented using Keras 1 with the Ten-sorFlow backend.
The two U-Net FCNs are implemented in a cascading way. 100 LiTS CT scans with various image dimensions are used for training. All training slices are resized to have common size, so the inputs for both FCNs are two-dimensional (2D) grayscale slices of size 256 × 256, and their outputs are binary mask images of size 256 × 256.
The first network is trained to segment the liver envelope in whole abdomen slices, which are resampled to input size (256 × 256), so that the network concentrates on learning features that discriminate liver from background. The second network is trained to segment the tumors, given the liver envelope image. The segmented liver from the first framework step is cropped and resized to the second network input. The liver ROI helps in reducing the percentage of misclassified nontumor pixels. The second U-Net FCN can concentrate on learning features that discriminate tumors from liver background segmentation.
The soft dice coefficient (DSC) is used as loss function that is computed on the pixel-wise softmax of the FCN final feature map.
Due to segmenting small objects like tumors, class balancing according to the pixel-wise frequency of each class in the data is required.
To deal with this case, the training datasets ensured to have the corresponding mask for each input 2D slice, so each batch contains patches where both tumor and background are present. In addition, to focus the model on the liver/tumor structure, the training process excludes slices that does not have corresponding mask.
Both networks are trained with 20 epochs (mini-batch size 32).
The network parameters are updated using Adam optimizer with 0.001 learning rate. The learning rate is reduced by factor of 0.1 F I G . 2. Effect of windowing and median filtering. The raw computed tomography slice (left) and the enhanced slice using median filtering (right). every 5 epochs to ensure a balanced loss, if no improvement in network optimization is acquired. Figure 5 shows the learning curves of the proposed FCNs for liver and tumors; the achieved validation accuracies for liver and tumor FCNs are 97.7% and 88.8%, respectively.

2.D | Localized region-based level set
Fully convolutional neural-predicted segmentation may not reach the liver or tumor boundaries in some test CT scan, since the FCN output accuracy is limited by the learned features from the training datasets. Level set-based active contour method is used in this work to refine the FCN-predicted segmentation, to match more accurately the boundary of the liver organ or the tumors inside the CT scan.
Active contour-based techniques have been widely used for image segmentation and boundary tracking. 18 The basic idea of active contour methods is to start with initial boundary shapes represented in a form of closed curves, that is contours, and iteratively allow the contour to deform so as to minimize a given energy functional according to the constraints of the image, in order to produce the desired segmentation. Level set-based active contour is a formulation to implement active contours that was proposed by Osher and Sethian. 23 Two main categories exist for level set active contours: edgebased and region-based. Edge-based active contour models utilize image gradients in order to identify object boundaries; however, this type has been found to be very sensitive to image noise and depend on the initial contour place. On the other hand, the region-based level set active contour has advantages compared to edge-based level set methods that include robustness against initial contour place and insensitivity to image noise. 24 Since the FCN-predicted segmentation is expected to be close to the liver/tumor boundary, the region-based level set active contour seems to be more suitable than other level set types, namely that proposed by Chan and Vese. 18 The Chan-Vese energy (Ecv), which is aimed to be minimized, is referred by Eq. (1): Computed tomography scan enhancement using edge enhancing diffusion filtering.
where Ωc represents the interior of the curve C, and c1 and c2 are the mean intensities for the interior and the exterior of the curve to be defined in an image I. The first term is the regularization term that minimizes the curve length s, and the second term maintains the balancing between the interior and the exterior. To make this step more efficient, the localized implementation of this active contour method is used. 25 Instead of modeling the region of the whole image, the curve is modeled by many neighborhood local regions, each local region is considered separately, which is divided into local interior and local exterior, as explained in Fig. 6. and 3459 for liver and tumors segmentation, respectively.
In order to demonstrate the robustness, generalization, and scalability of the proposed method, the proposed method is applied on 50 datasets from two publically available datasets, LiTS and IRCAD.
As mentioned earlier, 31 LiTS CT scans are devoted for testing and evaluation. Besides that, the 3D IRCAD dataset 3 is also used for testing and evaluating the proposed methods. IRCAD dataset has higher variety and complexity of livers and its tumors, and IRCAD dataset includes 20 venous phase-enhanced CT volumes acquired with different CT scanners. IRCAD datasets are pathological CT cases, which have 111 tumor cases residing inside the liver envelope.

3.B | Qualitative and quantitative results
The qualitative results of the automatic liver segmentation for two different examples are visualized in Fig. 7. Comparison with segmented liver using the proposed method, U-Net-predicted segmentation and ground truth, gives rise to the assumption that the proposed approach is highly promising to achieve high performance.
The U-Net computes the soft label probability maps. It examines each pixel of the test CT scan and assigns it to one of the two labels, liver or background for the liver segmentation step, and tumor or liver tissue for tumor segmentation step. The localized regionbased level set step deforms the U-Net output (liver or tumor) to match the boundary of the structure based on the intensity differences around the initial contour.
The proposed method succeeded to segment the liver organ from different CT scans that come with complex structures and different intensity homogeneities. In general, it can be observed from Automatic liver segmentation with fully convolutional neural networks U-Net and region-based level set. Green depicts correct liver segmentation (level set + U-Net). Blue for predicted liver segmentation (using U-Net) and red is the corresponding ground truth.
especially the contrast differences between tumors and liver parenchyma, from different training datasets to train the FCN network. This is followed by extracting the tumor intensity range of the target image using region-based level set step, which is used to refine the initial segmentation of the FCN-trained network. In this work, the 2D region-based level set method is used instead of the 3D version, that is because the 2D level set segmentation performs better than the 3D in terms of curve evolution on each individual slices.
As the framework consists of two consecutive steps, liver then tumor segmentation, the potential limitation of the proposed method is that accuracy of tumor segmentation relies on the liver organ segmentation step. The segmentation of the tumors is carried inside the extracted liver envelope from first step. It could happen in some datasets that liver parenchyma has similar intensity homogeneity F I G . 8. Automatic tumor segmentation from liver region of interest with U-Net FCN and region-based level set. Green depicts correct tumor segmentation (level set + U-Net), blue for predicted liver segmentation (using U-Net) and red is the corresponding ground truth.
T A B L E 1 Mean dice evaluation for automatic liver segmentation. with tumors that impose a challenge to extract accurate liver envelope. This work assumes that all CT scans are acquired at the portal venous phase of image acquisition, in which the tumors and liver parenchyma have a clear contrast.

Method IRCAD LiTS
The proposed method demonstrated the improvement of using level set technique and the use of local information in the target image to enhance the FCN-predicted segmentation and achieve accurate segmentation output. Based on the evaluation results, the proposed method achieved high segmentation quality in detecting liver and tumors from CT images. The proposed method succeeded to segment liver and tumors in heterogeneous CT scans from different scanners, as in IRCAD dataset, which proved its ability for generalization and be promising tool for automatic analysis of liver and its tumors in clinical routine.

CONF LICT OF I NTEREST
The authors declare that they have no conflict of interest.

E T H I C A L A P P R O V A L
All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.
For this type of study, formal consent is not required.

I N F O R M E D C O N S E N T
Informed consent was obtained from all individual participants included in the study.