2.1

Method overview

In this study, liver motion and liver tumor localization were solved via deformable registration between a liver mesh (extracted offline from a prior CT/CBCT image available before the treatment) and the liver features projected on a single X-ray projection (Fig. 1). The registration was achieved via two steps: (a) liver surface motion estimation via a deep graph neural network-based structure (Fig. 2); and (b) intra-liver motion estimation via biomechanical modeling. Specifically, in step (a) a patient-specific DL model was trained to predict a liver boundary deformation vector field (DVF) that deforms the prior liver surface mesh to match with the liver shape variations encoded in the X-ray projection. In step (b), a biomechanical model of the liver was built, and an intra-liver DVF was solved through finite element analysis using the liver boundary DVF as the boundary condition.

Figure 1.

Workflow of the proposed method. A deep graph neural network-based model was trained to predict liver surface deformation vector field (DVF) from a single X-ray projection. Then a biomechanical model solves the intra-liver DVF using the liver surface DVF as the boundary condition for tumor localization.

Figure 2.

Overview of the deep-learning (DL) network that estimates liver boundary motion from an on-board X-ray projection. The network consists of two subnetworks performing feature extraction and liver boundary DVF prediction separately. The first subnetwork uses ResNet-50 to extract image features from an X-ray projection. The extracted feature maps were pooled for each node of a liver surface mesh by the perceptual feature pooling layer, based on the projected node coordinates on the X-ray projection. The second subnetwork, consisting of three deformation blocks, progressively estimates liver boundary DVFs. A deformation block comprises of a graph convolutional network (GCN) and a spatial transform layer. The GCN was learned to predict a liver boundary DVF based on the features extracted from the ResNet-50 subnetwork. A spatial transform layer deforms the prior reference mesh or the deformed liver surface mesh from the previous deformation block, using a GCN-predicted DVF.

2.2

The deep-learning network architecture

The network was trained to learn image features from on-board X-ray projections to predict boundary movement from the prior liver mesh to each on-board projection. The DL network architecture is illustrated in Fig. 2, which contains two subnetworks. The first subnetwork extracts image features from each on-board X-ray projection, and the extracted feature maps are pooled and fed into the second subnetwork for liver boundary DVF prediction. Here we used ResNet-50 [8] as the feature extraction network. Consisting of a series of convolutional layers stacked in a residual learning architecture, ResNet-50 extracts encoded liver shape variations, via local and global image features contained in the X-ray projection, and learns short- and long-range dependencies among these extracted features. These learned dependencies are helpful for the deformation estimation because they are shown common in respiration-induced liver motion. The perceptual feature pooling layer pools the ResNet-50 extracted feature maps by associating each 3D node of a liver surface mesh with a 2D point in the feature maps, based on the same geometry of the cone-beam projection.

The second subnetwork comprises a series of deformation blocks that progressively deforms the liver surface mesh nodes based on the extracted feature maps from the first subnetwork. Each deformation block involves a graph convolutional network (GCN, Fig. 3) and a spatial transform layer that deforms the liver surface mesh using the GCN-predicted DVF [9]. GCN performs graph-based convolutions that generalize the standard convolution operations to data structures lack of underlying Euclidean structures, such as functional networks in brain imaging. A non-Euclidean data structure can be represented by a weighted graph comprised of a set of vertices, edges connecting the vertices, and weights associated with each vertex (e.g., vertex features, DVFs, vertex-associated image features). The use of GCN is indicated for our problem, as the liver surface mesh nodes, the geometrical connectivity (edges) between the nodes, and the learned image, DVF, and vertex features associated with each node make a standard non-Euclidean data structure for inputs into the GCN. Using extracted image features that encode the liver shape, preceding DVFs, and learned vertex features from the previous block, the GCN learns to predict a liver surface DVF to further deform the surface mesh deformed by the previous block. The inputs of the GCN in the first deformation block contain ResNet50-extracted image features and an initial DVF which was set to be zero. For each subsequent GCN, the image features were re-pooled based on the new node coordinates (Fig. 2), deformed via the spatial transform layer and the DVF predicted by the preceding block. The image features were then input into the subsequent GCN, along with the predicted surface DVF and the learned vertex features from the preceding block. We used a GCN of the same architecture as the G-ResNet [9], which is illustrated in Fig. 3. The corresponding network was modified and adapted from the Pixel2Mesh library [10]. The model training was driven by a loss function involving a mesh similarity loss and regularization losses that regularize the deformation and enforce smoothness of the boundary DVFs.

Figure 3.

Graph convolutional network (GCN). The inputs contain pooled image features from the feature extraction ResNet-50 subnetwork (Fig. 2), a surface DVF, and vertex features yielded from the GCN in the previous deformation block (if any). The GCN consists of 20 graph convolution layers that, except for the entrance and exiting layers, were organized in a residual learning architecture. The GCN yields a surface DVF and vertex features to feed into the subsequent deformation block. The inputs of the first GCN in the second subnetwork contains only image features and an initial surface DVF which was set to be zero. The image features were re-pooled for each GCN based on deformed node coordinates. The rounded box in the middle represents a residual learning module containing three graph convolution layers with a shortcut connection, which iterates 6 times.

2.3

Biomechanical modeling

After the deep neural network solves a liver boundary DVF to match with the liver shape features on the X-ray projection, the intra-liver DVF was subsequently derived using a biomechanical model. Here we used the Mooney- Rivlin material model, which describes a hyperelastic (i.e., nonlinear elasticity) material that fits biological tissues well. The details of implementing the biomechanical model can be found in Ref. [6].

2.4

Dataset curation and augmentation

A dataset of 10 patients with liver cancer from our institute was used to evaluate the proposed method. The study was approved under an institutional review boards protocol. Each patient had a contrast-enhanced four-dimensional CT set from treatment planning, and the CT images were binned into 10 respiratory phases (from 0% to 90%), with 0% being the end-of-inhale phase. The CT images were resampled to a uniform size of 256×256×128 with an isotropic resolution of 2 mm×2 mm×2 mm. On-board X-ray cone-beam projections were simulated from the CT images using a ray-tracing algorithm. We simulated projections from three angles: 0, 45, and 90 degrees. The 0- and 90-degree are for anteriorposterior and left-right directions, respectively.

Since each patient had only a 10-phase 4D-CT set, to generate sufficient motion variation scenarios to train the patient-specific network and avoid overfitting, we augmented the dataset of each patient by simulating realistic respiratory deformations encountered in on-board liver imaging. The augmentation was based on a PCA-based motion model of each patient [6, 11]. We first performed deformable registrations between the reference 0% phase and the other phases to attain DVFs, using the open-source software package Elastix. To improve the intra-liver DVF accuracy, we applied biomechanical modeling to derive intra-liver DVFs, using the liver surface DVFs solved by Elastix as boundary conditions. We then replaced the Elastix intra-liver DVFs with the biomechanical modeling-derived intra-liver DVFs. PCA was subsequently performed on these high-quality DVFs of each patient to obtain patient-specific principal motion components. For augmentation, the coefficients of the first three principal motion components were randomly scaled to re-generate DVFs of various magnitudes and patterns [11]. In total, for each patient we generated 1,728 augmented samples which were partitioned into training, validation, and testing sets. The partitioning was assigned according to the original respiratory phases of the PCA coefficients prior to the random scaling. The training set includes the samples of which the original PCA coefficients were from the 10% to 40% phases; the validation set includes the samples whose original PCA coefficients were from the 60% and 70% phases; and the testing set includes the samples whose original PCA coefficients were from the 50%, 80%, and 90% phases. 50% is the end-of-exhale phase that has the largest deformation from the 0% phase.

2.5

Evaluation schemes

The deformation accuracy of liver surface meshes was evaluated using the Hausdorff distance (HD) between the deformed and the ‘ground-truth’ target liver surface meshes extracted from the augmented dataset [12]. To evaluate the performance of liver tumor tracking, we manually contoured the tumors from the prior CT images at phase 0%. The tumor contours at 0% phase were then propagated using the augmentation DVFs (II.D.) to other augmented motion states, which were used as the ‘ground-truth’ to evaluate the ones deformed by our method. The accuracy of liver tumor tracking was evaluated by the Dice similarity score (DSC), center-of-mass error (COME), and HD.

3.1

Liver deformation accuracy

Figure 4 presents a qualitative comparison of liver surface meshes and projected nodes on X-ray projections at three projection angles (0, 45, and 90 degrees). The first row shows the surface mesh overlays between the prior and ‘ground - truth’ target meshes (left panel), and between the graph network-deformed and ‘ground-truth’ target meshes (right panel). The prior and target meshes correspond to the end-of-inhale phase and the end-of-exhale phase (with motion augmentation), respectively. The other rows show the overlay of the pre- (left panel) and post-registration (right panel) surface mesh nodes onto the corresponding X-ray projections of the end-of-exhale phase, at three different angles. Both the surface mesh overlay and the node-projection overlay demonstrate high registration accuracy.

Figure 4.

(Firstrow) Liver surface overlays between the prior and ‘ground-truth’ target meshes (left) and between the graph network-deformed and target meshes (right). The yellow meshes are the target meshes corresponding to the end-of-exhale phase after augmentation, and the red meshes correspond to the prior (left panel) and deformed (right panel) meshes. (Otherrows) Liver surface nodes projected on X-ray projections at three projection angles. Left and right columns show the projected nodes corresponding to the prior and deformed surface meshes, respectively.

Table 1 summarizes the mean (±s.d.) liver HDs of the proposed method and the PCA-based 2D-3D registration method [7]. The DL-based method results in much smaller HDs than the PCA-based 2D-3D method.

Table 1.

Mean (±s.d.) liver Hausdorff distances.

3.2

Liver tumor tracking accuracy

Table 2 summarizes the mean (±s.d.) liver tumor DSCs, COMEs, and HDs of the proposed method at three projection angles. In addition, the results of PCA-based 2D-3D registration and diaphragm tracking are also presented in the Table for comparison. The diaphragm tracking is only able to localize the diaphragm in 2D from a single X-ray projection, thus we only used it to represent liver tumor motion along the superior-inferior (SI) direction. The COME of the diaphragm-based method is thus only for SI direction, with 3D COME potentially being much larger. Table 2 clearly shows that the proposed method has the best liver tumor localization accuracy, and the performance is consistent among different angles.

Table 2.

Mean (±s.d.) liver tumor DSC, COME, and HD. The COME for the diaphragm-based method is for the superiorinferior direction only (*).