Inter/intra-frame constrained vascular segmentation in X-ray angiographic image sequence

Background Automatic vascular segmentation in X-ray angiographic image sequence is of crucial interest, for instance, for better quantifying coronary arteries in diagnostic and interventional procedures. Methods A novel inter/intra-frame constrained vascular segmentation method is proposed to automatically segment vessels in coronary X-ray angiographic image sequence. First, a morphological filter operator is applied to remove structures undergoing the respiratory motion from the original image sequence. Second, an inter-frame constrained robust principal component analysis (RPCA) is utilized to remove the quasi-static structures from the image sequence. Third, an intra-frame constrained RPCA is employed to smooth the final extracted vascular sequence. Fourth, a multi-feature fusion is designed to improve the vascular contrast and the final vascular segmentation is realized by thresholding-based method. Results Experiments are conducted on 22 clinical X-ray angiographic image sequences. The global and local contrast-to-noise ratio of the proposed method are 6.6344 and 4.2882, respectively. And the precision, sensitivity and F1 value are 0.7378, 0.7960 and 0.7658, respectively. It demonstrates that our method is effective and robust for vascular segmentation from image sequence. Conclusions The proposed method is effective to remove non-vascular structures, reduce motion artefacts and other non-uniform illumination caused noises. Also, the proposed method is online which can just process one image per time without re-optimizing the model.


Background
Nowadays, coronary artery disease (CAD) is greatly threatening human health [1]. Since X-ray angiography (XRA) has better imaging quality and faster imaging speed, it is regarded as the gold standard for the diagnosis and treatment of CAD. However, due to the perspective projection of 3D anatomic structures, much 3D information has been lost and different anatomical structures are overlapped in the XRA images. Moreover, the injection of contrast agent and the blood flow variation bring in-homogeneous intensity of coronary artery. And the cardiorespiratory motion and patient movement also introduce motion artefacts to the XRA images. To improve the image-guided diagnosis and interventional procedures of CAD, the automatic and robust vessel segmentation is of great significance and meanwhile a challenging problem.
Vascular segmentation technique can be divided into two classes, including the model-based and learningbased methods. Based on the spatial continuity of vessels, level-set and active contour-based methods are commonly utilized. Wang et al. [2] utilized the level-set algorithm to segment the coronary artery by constructing the speed function with the curvature, intensity and model term. Sun et al. [3] proposed the local region based active contour method by shape fitting the energy function. It improved the segmentation accuracy and is much more robust to the nonuniform intensity distribution and fitting initialization. Based on the shape of vascular section, Cheng et al. [4] employed a B-snake model to accurately segment the small-scale vessels in the low-contrast images. Lee et al. [5] utilized the Kalman filter to initialize the contour and segmented the vessels by the active contour model. The initialization improved the time efficiency. The learning-based methods usually compute the classification model based on the image hidden information. Hassouna et al. [6] modeled the background with two Gaussian and a rayleigh distributions and the vessels with a Gaussian distribution, respectively. Then they utilized the Expectation-Maximization algorithm to estimate the distribution parameters and employed the Markov Random Field to be the spatial constraint to realize the final vascular segmentation. Goceri et al. [7] clustered the vessels based on the K-means approach and improved the segmentation accuracy by the morphology based iterative optimization. Lupascu et al. [8] delivered the high order features to the AdaBoost classifier to speed the segmentation. Orlando et al. [9,10] computed the fused feature map by the Fully-Connected Conditional Random Field to ensure the continuity of different vascular segments. In recent years, Convolutional Neural Network (CNN) based vascular segmentation has attracted much researcher's attention. Wang et al. extracted the vascular features based on CNN to segment the vessels with a stochastic decision forest. Fu et al. [11] combined CNN with Conditional Random Field (CRF) and developed a DeepVessel network to improve the segmentation accuracy. Luo et al. [12] improved the DeepVessel network by considering the non-uniform intensity and noise coexistence.
To the authors' knowledge, the model-based segmentation methods are sensitive to the initial contour and learningbased approaches require large amounts of labeled datasets. Moreover, the methods mentioned above have a significant limitation in angiographic images with low contrast and noisy background. Vascular enhancement can greatly ease the vascular segmentation by enhancing the vascular structures and compress the background noise. The single-image based enhancement easily introduces the non-vascular noise and motion artefacts when dealing with X-ray angiogram images. While subtraction-based enhancement utilizes the angiograms with and without vessels. It can effectively remove the motion artefacts in the final enhanced vascular angiograms and improve the subsequent segmentation accuracy.
Current vessel subtraction methods can be classified into two categories, including image registration based methods and layer separation based methods. In the imaging of coronary artery, mask images are taken prior to the perfusion of the contrast agent and coronary arteries are not visible in them. While live images are taken during the contrast agent passing through the coronary artery. Image registration based methods [13,14] only need a live image and a mask image whose motion is the most similar to the live image. Such methods are usually realized by template matching, similarity measure maximization, image warping and subtraction technology sequentially. Though the technique largely reduces the motion artifacts and non-vascular noise, it is likely to be interrupted by the patient motion or contrast agent leakage when computing the correspondences between images. Motion layer separation based method supposes an image in the sequence can be decomposed into motion layers. The key part of the first type methods [15][16][17] is motion estimation of each layer. Zhu et al. [15] divided the sequence into the vascular and non-vascular layers and applied optical flow to the non-vascular layer to compute the deformation filed. Zhang et al. [16] separated the sequence into three layers, including static, lung (slow motion) and vessel (rapid motion) layer and constructed a motion transformation model for each layer. Nevertheless, the structures in the XRA sequence participate in different motion patterns. Specific motion model in each layers cannot cover all the motions of a structure, especially vascular motion in the XRA sequences includes the cardiac, respiratory, patient and camera motions. Another kind of methods [18][19][20] supposes that the image is under specific prior constraint and directly separates the sequence into background and vessel layer. Many mathematical expressions, such as L1 norm, L2 norm, nuclear norm and so on, have been applied to model the specific prior. Robust principal component analysis (RPCA) model, composed of sparse and low-rank prior, has become a common tool in medical image analysis of various imaging modalities.
In this paper, we propose an inter/intra-frame constrained vascular segmentation in the angiographic image sequence. First, a morphological filter operator is applied to remove motion artefacts caused by respiratory motion from original XRA sequence. Second, an inter/intra-frame constrained RPCA (IFC-RPCA) is utilized to extract the vascular images. Third, a multi-feature fusion is designed to realize the final vascular segmentation. The proposed method is effective to remove non-vascular structures, reduce motion artefacts, other non-uniform illumination caused noise and preserve the local information of vascular structures.

Methods
In this section, the vascular images is distinguished from the XRA sequence by removing the structures that are static or undergoes the respiratory motion. Then, a multi-feature fused descriptor is employed to further compress the background noise in the vascular images and the vascular structures are finally segmented by a thresholding-based approach.
Inter/intra-frame constrained RPCA To extract the vascular images from the XRA sequence I, structures that are static or undergo the respiratory motion should be removed. To reduce the disturbance (lung, diaphragm) caused by the respiratory motion, a circular structural element based morphological close operation [18] is applied to the sequence I to obtain the respiratory sequence R. By subtracting the sequence R from sequence I, the respiratory disturbance can be removed and the obtained sequence is denoted as DI. The sequence DI is composed of two components, including the moving vascular component and the quasi-static non-vascular component. In addition, the vessels in sequence DI only occupy a small portion. Considering RPCA aims to decompose the matrix into a low-rank component and an overall sparse component by searching for a low-dimensional subspace, it is suitable to separate sequence DI into the moving vascular component and the quasi-static non-vascular component [21]. Hence, we have: where E refers to the vascular component, and B is the quasi-static non-vascular component. β 1 and β 2 are regularization coefficients. ‖•‖ F is the Frobenius norm, ‖•‖ * is the nuclear norm and ‖•‖ 1 is L1 norm.
Considering dealing with the steaming X-ray images for coronary interventions, the online processing of XRA sequence is essential on the basis of the motion information in inter-frames. Hence, we utilize the explicit lowrank factorization [21] to describe B by the subspace basis Lr and the corresponding coefficients Ce. The factorization can be denoted as B = Lr × Ce T . After this, solving Eq. (1) equals to minimize the empirical cost function g(Ce, E, Lr), and we have: However, due to the non-rigid motion between the frames in the XRA sequence, large amount of motion artefacts and noises may still exist in the vascular component. Hence, we utilize the same morphological close operation to remove the motion artefacts around the catheter and obtain another difference sequence DI ′ . To remove more motion artefacts and noises, we introduce the intra-frame constrained RPCA and denote it according to Eq. (2) as follows: where Lr In addition, since most static structures have been removed in the inter-frame constrained RPCA, the optimization of intra-frame constrained RPCA will not depend on the motion information across the image sequence DI ′ . Hence, we utilize a 1 matrix as Lr 0 i and the optimization of intraframe constrained RPCA only need to update fCe 0 i ; E 0 i g, as follows: Through the intra-frame constrained RPCA, the final enhanced vascular sequence V is obtained. In sequence V, the contrast of vessels in the images is improved and the background is smooth and clean.

Multi-feature fused vascular segmentation
For each image V i in V, we utilize VI i (x) to represent the intensity of i th image in VI at x, and x = [x 1 , x 2 ] T which refers to the pixel coordinate. The Hessian matrix at scale σ can be computed as follows: We use λ 1 and λ 2 to be the eigenvalues of matrix H, and v 1 and v 2 to be the corresponding eigenvectors of matrix H. For the pixels belong to the vascular structures, the eigenvalues should satisfy the principle |λ 1 | ≈ 0, |λ 1 | ≪ |λ 2 |. The directions of eigenvectors v 1 and v 2 are along with the vascular centerline and perpendicular to the vascular tangential direction, respectively.
Since the vascular structures contain the elongate and round-sectional segments (bending, bifurcations and diseased vascular segments), a good vascular feature descriptor should distinguish the vascular segments with other structures. Hence, we design a new vascular feature descriptor. To avoid the compression of feature descriptor, the first feature descriptor [22] F 1 (x, σ) at each pixel is computed as follows: However, F 1 (x, σ) has a non-uniform response when the bending and bifurcation of vascular segments appear and is easily changed by the non-uniform illumination introduced by the contrast agent infusion. To avoid these conditions, another feature descriptor [23] F 2 (x, σ) at each pixel is calculated as follows: And, But F 2 (x, σ) appears serious blurring when the vascular segments are overlapped or very close to each other. Hence, to improve the vascular contrast and compress the non-vascular structures, we fuse the two feature descriptor with a weighted pattern to produce the uniform response of vascular segments and improve the boundary accuracy of vascular segments: Until now, we obtain a feature value for each pixel in image VI i . Since the new feature descriptor can effectively distinguish the vascular and non-vascular pixels, the segmented vascular structure image SI i can then be obtained from image VI i by only utilizing a threshold value.

Experimental results
All the experiments were implemented in MATLAB (The MathWorks, Inc.) under the Windows 10 environment, and all the experiments were conducted on a relatively low-cost PC with 16 GB RAM and 3.2 GHz Intel CPU.

Dataset and evaluation criteria
The proposed method was evaluated on 22 XRA sequences collected from the Peking Union Medical College Hospital. The size of all the images in the sequences is 512 × 512 and the resolution of each image is 0.3 × 0.3 mm 2 . In all 22 XRA sequences, the inflow and wash out of contrast agent To quantitatively evaluate the proposed IFC-RPCA method, two kinds of masks are generated. One is used to evaluate the global vascular contrast and pixels within and outside the annotated vascular region consist of the vessels and background, as shown in Fig. 1b. The other one is used to evaluate the local vascular contrast. As shown in Fig. 1c, vascular region is the same with the global mask, and the background region is comprised of the pixels in the white region, which is the 7-pixel-wide neighborhood of the vascular region boundary.
To evaluate the performance of the proposed method, it will be compared with the multiresolution elastic registration (MER) method [14], the online robust principal component analysis (ORPCA) method [19], the graduated RPCA with motion coherency constraint (MCR-RPCA) method [20]. To evaluate the proposed segmentation method, it will be compared with Fully-Connected Conditional Random Field (FC-CRF) method [10] and level-set-based method (LevelSet) [24].
Contrast-to-noise ratio (CNR) [18] is utilized to evaluate the vascular contrast of the vessels and can be defined as follows: where μ F and μ B are the mean gray values in the vascular and background regions in the extracted vascular images. σ B is the standard deviation of the gray values in the background region. We compute the global CNR and local CNR based on global mask and local mask, respectively.
To evaluate the proposed segmentation method, we utilize five metrics including the precision (pre), sensitivity (sen) and F1 value. In addition, the metrics are computed as follows: where TP, FP and FN indicate the true positive (correctly identified vessel pixels), false positive (incorrectly identified vessel pixels) and false negative (incorrectly identified background pixels), respectively.

Results
In our experiments, all the parameter settings are empirical. In detail, diameter of the circular structural element ; M2Þ , r = r ′ = 5, p = 0.3, α 1 = α 2 = 0.5, M1 and M2 where are the size in each dimension of an image. Figure 2 shows the extracted vascular images by the proposed IFC-RPCA method. The order numbers of the three random selected images in the first two rows are 17th, 23th, 31th. The order numbers of images in the last two rows are 19th, 24th and 41th, respectively. According to the order, the inflow of contrast agent is gradually infused within the coronary artery. In Figs. 2(a1) and (c1), the contrast agent is not fully infused within the coronary artery. In Figs. 2(a2) and (c2), vessels are in the diastole stage, while in Figs. 2(a3) and (c3), vessels are in the systole stage. As can be seen from the extracted vascular images, the vascular structures are preserved throughout the XRA sequences and present a very high contrast. In addition, motion artefacts and other non-vascular noise are also removed and vascular segments with small scales are also preserved.  Table 1 compares the global CNR and local CNR over all the annotated XRA images by four different methods, including MER, ORPCA, MCR-RPCA and IFC-RPCA, respectively. All the methods obtain larger global CNRs than the original images and greatly improve the  contrast of vessels. The global CNR of IFC-RPCA is much larger than MER and yields 17.37 and 37.91% improvement by comparing with ORPCA and MCR-RPCA, respectively. The performance of IFC-RPCA is achieved by removing the artefacts near the vessels, catheters and non-vascular noise. For the local CNR, the values by MER is smaller than the local CNR of original images which demonstrates that MER cannot improve the contrast within perivascular regions. IFC-RPCA obtains 10.20 and 47.46% improvement by comparing with ORPCA and MCR-RPCA, respectively. The proposed IFC-RPCA can also make the boundaries of the vascular structures much clearer.
We also simulate the angiograms with low dose contrast agent which has significant clinical value for the clinicians and patients. The simulated images are generated by linearly subtracting the enhanced vascular image from the original image. Figure 4 shows an example of the simulated image with low dose contrast agent. As can be seen from Fig. 4(c), the contrast of the vessels is greatly reduced. Figure 5 shows the subtraction results based on the simulated images with low dose contrast agent. In Fig. 5a, it is very difficult to distinguish the vascular structures from the background. While in Fig. 5c, the intensity of the vessels is very close to the bones. As can be seen from Figs. 5b and d, the proposed IFC-RPCA method can improve the contrast of the vessels and meanwhile, remove the diaphragm and bones. Fig. 6 shows the segmented vascular structures by the proposed method. Angiograms in the first column are randomly selected from image sequences. Images in the third column are computed by the proposed feature descriptor. In the images, there are uniform responses when the vascular segments appear bifurcation, overlapping or are very close to each other. In addition, the responses in vascular regions are much larger than the background which brings the vessels high contrast. Images in the fourth column refer to the segmentation results by a threshold value from images in the third column. In the images, the vascular edges are preserved even when different vascular segments are very close. In addition, vascular segments with small scales are also accurately segmented. In the fifth column, vascular segments in green color refer to over-segmentation, while vascular segments in blue color refer to undersegmentation. As can be seen from the figures, the vessels will be fractured when the intensity of vascular segments is close to the background. For the vessels with large scales, they have precise boundaries and are (a4)-(c4) segmented results; (a5)-(c5) color map between the ground truth and segmented results. Red color: correctly identified vessel pixels, green color: incorrectly identified vessel pixels and blue color: incorrectly identified background pixels consistent with the ground truth. As also can be seen from the segmented results, the non-vascular noise is almost removed. Figure 7 shows the comparison results by methods, including LevelSet, FC-CRF and the proposed model. In the first column, two right and two left coronary artery angiograms are randomly selected from the image sequences. In the third column, vascular segments with large scales are precisely segmented and vascular segments with small scales present serious noise. In addition, LevelSet method is semi-automatic and requires the manual labeled seed points. In the fourth column, large amounts of noise appears in the non-vascular regions and there are many holes in the vascular segments. After comparing with the ground truth, the diameters of the obtained vascular segments are smaller than the actual vessels which make a great influence to the subsequent parameter measurement of vascular segments. In the fifth column, there are fractures when the vascular segments appear small-scales. But the vascular segments with large scales are precisely segmented. In addition, when the vascular segment is overlapped with the diaphragm, the segmentation results doesn't introduce motion artefacts. For each angiogram in the figures, the segmentation results by the proposed method present a clean background without non-vascular noise. Table 2 provides the quantitative comparison in pre, sen and F1 value by LevelSet, FC-CRF and the proposed method. Pre, sen and F1 value of the proposed method are all the largest, the vessels and noise are both disposed best. All the metrics of LevelSet are all superior to those of FC-CRF. The vascular structure preservation and background noise removing are all better than these of FC-CRF.

Conclusion and discussion
In the paper, we propose an inter/intra-frame constrained vascular segmentation method and demonstrate its application in the XRA sequences. Experimental results demonstrate the effectiveness of the proposed method in accurate vessel segmentation in the XRA sequence. As can be seen from the experimental results, the proposed IFC-RPCA effectively reduces the lung tissues, diaphragm and vertebral bodies and removes the motion artefacts near the catheter and non-vascular noises from the XRA sequence. The proposed IFC-RPCA yields 17.37 and 37.79% improvement in global CNR and 10.20 and 47.46% improvement by comparing with ORPCA and MCR-RPCA methods. The proposed multi-feature fused feature descriptor produces uniform response in different vascular segments and makes the vascular structures high contrast with the background. Based on this, the vascular structures can be simply segmented with only a threshold value. We obtain 0.7378, 0.7960 and 0.7658 with respect to the precision, sensitivity and F1 value, respectively. It demonstrates the proposed method can both effectively dispose the vessels and background. The proposed vessel segmentation method is online without re-optimizing the whole model and automatic, it is very suitable to be applied in the intra-operative image guided surgical navigation.