Keywords

1 Introduction

Image registration is one of the central tasks in medical imaging with a wide range of application. The overall goal is alignment of images by spatially mapping corresponding locations. Registration typically stands at the beginning of an image processing pipeline and once spatial correspondence has been established, it allows for subsequent local or even voxel-wise comparison or other local processing. Typical usage in medical imaging is navigation support, motion correction, propagation of information such as markers or segmentations, change detection and change analysis. Behind these examples, there are hidden two competing registration goals that generally cannot be reached simultaneously. While the first examples ask for local alignment as perfect as possible, the assessment of (in particular) morphological changes requires to keep local differences. Thus, finding the right balance is a challenging and application depending task that needs to be solved by every image registration method.

In this work we present an approach dealing with this issue in the context of software support for reading and analyzing thorax-abdomen CT scans that undergo an oncological workup. Here, accurate deformable image registration can be used to compare follow-up scans by synchronized viewing, to link courser positions to the retrieval and to propagate findings from prior images in follow-up scans. On the other hand, if the registration keeps local differences, it can be used for subtraction imaging with baseline and warped follow-up scans to assist radiologists with the detection and quantification of changes, such as new lesions or tumor growth.

State-of-the-art deformable registration approaches try to achieve a reasonable trade-off between alignment quality and preservation of local changes. For example in a variational registration setting, this is typically steered by a so-called regularization parameter that weights image similarity versus smoothness of the computed mapping [2, 5, 9, 12, 16, 21]. Also variational approaches were introduced for deformable registration that incorporate local rigidity. This is done either by adding an penalty term to the objective function to be minimized [8, 10, 11, 17, 18] or by forcing local rigidity as hard constraints [6, 7, 13, 14]. However, all these methods have been proposed for modeling stiff tissue such as bone and they require prior knowledge about the regions that shall be kept rigid. In our setting we generally cannot assume to have such prior information available. Furthermore, utilizing these type of methods would be quite costly since we have to run a complete registration if we change the local region to be kept rigid. To this end, we follow the ideas of Dzyubachyk et al. [3]. The authors introduced an interactive method with focus on finding bone lesion in follow-up MRI scans. Therefore, the user selects a point of interest on a skeletal structure, the surrounding area of interest is segmented by region growing and a locally rigid transform is derived from a pre-computed whole-body deformable registration. The derived rigid transformation is then used for visual side-by-side comparison of follow-up MRI scans by a lens view, color-fusion, warped iso-contours and a quiver plot of the local deformation.

In this work we extend the ideas from [3] to CT follow-up imaging, change detection and subtraction imaging. We consider a generalized setting for taking full advantage of locally precise deformable registration. We also present a lens tool called rigid lens for visualizing, detecting and analyzing changes by locally rigid approximations of the deformation field. We give quantitative measures for rating and detection of changes and evaluate the rigid lens and its use for change detection and change assessment with a quantitative study based on 1492 annotated tumors in thorax-abdomen CT follow-up scans.

2 Method

Our idea follows the work presented in [3] and is inspired by common lens viewing tools for interactive inspection of changes and image fusion of aligned images where one image is shown in the background and another image is shown inside a lens region. We assume that we have given two registered images which we call reference and template image and the corresponding deformation vector field warping the template image onto the reference image. Then in principle we could use a common lens tool to inspect the reference and warped template. If the registration produced a reasonable alignment we will not recognize significant morphological differences between inside and outside the lens, since local change have been removed by registration. In the extreme case of perfect alignment the warped image will look almost identical to the reference image. However, our idea is to compute a rigid approximation to the given non-rigid deformation restricted to the lens region. Then we use the obtained rigid mapping for warping the template and show the result inside the lens region. As a result we obtain a locally rigid registration valid for the particular position of the lens region with complete morphology from the template preserved. The concept of the rigid lens is illustrated in Fig. 1 and the details are given next in Sects. 2.1 and 2.2.

Besides using the rigid lens for interactive viewing, where users hovers with the rigid lens over the reference image, we are also interested in its use for change detection. The most simple extensions is to perform subtraction imaging between reference and rigid lens. However, we are also interested in deriving measures that can be used for automatic change detection. To this end, in Sect. 2.3 we present three measures based on the hypothesis that relevant changes alter shape, size or appearance of structures.

Fig. 1.
figure 1

Schematic overview of the rigid lens concept.

Full size image

2.1 Modeling

Let \(R,T:\mathbb {R}^3\rightarrow \mathbb {R}\) denote the reference and template image, respectively, and let \(\varOmega \subset \mathbb {R}^3\) be a domain modeling the field of view of R. Then the goal of image registration is to find a deformation \(y:\varOmega \rightarrow \mathbb {R}^3\) that aligns the reference R and template T such that R(x) and T(y(x)) are similar for \(x\in \varOmega \). For example common variational registration approaches compute y as a minimizer of an objective function of the type

$$\begin{aligned} \mathcal {D}(R,T(y))+\alpha \mathcal {S}(y) \end{aligned}$$

with so-called distance measure \(\mathcal {D}\) that quantifies the similarity of reference R and warped moving image T(y), smoother \(\mathcal {S}\) that forces smoothness of the deformation and a regularization parameter \(\alpha >0\) that weights smoothness versus similarity. However, in the following we just assume that y is a non-rigid deformation produced by some registration algorithm and that reasonably well aligns R and T. Furthermore, for practical purpose we restrict ourselves to the discrete case. To this end, we assume the domain \(\varOmega \) is discretized by a uniform grid with resolution \(h>0\) and let \(\varOmega _h\) be the set of all cell-centered points. The idea of the rigid lens is to locally approximate y by a rigid transformation \(y_{\text {rigid}}\) on a lens region \(\mathcal {L}\) defined as a neighborhood of a point \(x_0\) and radius \(r>h\):

$$\begin{aligned} \mathcal {L}\equiv \mathcal {L}_r(x_0) := \{ x \in \varOmega _h \,:\, \Vert x-x_0 \Vert _2 \le r \}. \end{aligned}$$

For ease of notation, in the following we denote the lens region just by \(\mathcal {L}\) always with the implicit understanding that \(\mathcal {L}\) depends on center \(x_0\) and radius r. Then we define \(y_{\text {rigid}}\) as rigid least squares solution, such that

$$\begin{aligned} \sum _{x \in \mathcal {L}}^{} \Vert y(x) - y_{\text {rigid}}(x) \Vert ^2 {\mathop {=}\limits ^{!}} \min . \end{aligned}$$
(1)

Note that \(y_{\text {rigid}}\) depends on the lens region and center \(x_0\), radius r, respectively, i.e., \(y_{\text {rigid}}(x)\equiv y_{\text {rigid}}(\mathcal {L};x)\equiv y_{\text {rigid}}(\mathcal {L}_r(x_0);x)\).

2.2 Algorithm

Least-squares-estimation of rigid transformations for fitting point clouds is a well-known problem in computer vision. Problem (1) is also known as Procrustes matching and tracing back to mid 1960’s to the work of Whaba [20] and Schönemann [15]. Since then, various algorithms and methods have been proposed [4]. Dzyubachyk et al. [3] uses a unit quaternions based approach. Here we follow the work of Arun et al. [1] and Umeyama [19], that is based on the singular value decomposition which has been shown to be the numerically most stable method [4]. For sake of completeness we give a sketch of the algorithm.

The rigid transformation can be parameterized by a rotation matrix \(Q\in \text {SO}(3)\) and translation vector \(b \in \mathbb {R}^3\), such that \(y_{\text {rigid}}\) can be written as \(y_{\text {rigid}}(x) = Qx + b\) and above least-squares problem is equivalent to find Q, b such that

$$\begin{aligned} \sum _{x \in \mathcal {L}} \Vert y(x) - (Qx+b) \Vert ^2 {\mathop {=}\limits ^{!}} \min \ \text { s.t. } \ Q^{\top }Q = I \text { and } \det (Q) = 1. \end{aligned}$$
(2)

First we compute mean and covariance of the point sets \(\mathcal {L}\) and \(y(\mathcal {L})\). We set

$$\begin{aligned} \mu _x := \frac{1}{|\mathcal {L}|}\sum _{x\in \mathcal {L}} x, \quad \mu _y := \frac{1}{|\mathcal {L}|}\sum _{x\in \mathcal {L}} y(x) \quad \text {and} \quad \varSigma _{xy}:= \frac{1}{|\mathcal {L}|}\sum _{x\in \mathcal {L}}( x - \mu _x)(y(x) - \mu _y)^\top . \end{aligned}$$

Next we compute the singular value decomposition \(\varSigma _{xy} = UDV^\top \), with the diagonal matrix \(D = \text {diag}(d_1,d_2,d_3)\) and singular values \(d_1 \ge d_2 \ge d_3 \ge 0\). If \(\text {rank} (\varSigma _{xy}) \ge 2\) then (2) has a unique solution

$$\begin{aligned} Q^* = USV^\top \quad \text { and } \quad b^* = \mu _y - Q^*\mu _x, \end{aligned}$$

with diagonal matrix S defined as

$$\begin{aligned} S :=\left\{ \begin{array}{ll} I, &{} \text { if } \det (U)\det (V) = 1,\\ \mathrm {diag}(1,1,-1),&{} \text { if } \det (U)\det (V) = -1. \end{array} \right. \end{aligned}$$

Thus the solution \(y_{\text {rigid}}\) of (1) is given by

$$\begin{aligned} y_{\text {rigid}}(x) = Q^*x + b^*. \end{aligned}$$

Note that from practical perspective we expect \(\varSigma _{xy}\) having full rank as this is the case iff \(\mathcal {L}\) contains at least three (linear independent) points and the deformation y is invertible. Otherwise, either the lens region is degenerated or the registration results will most likely cause locally non-feasible deformations such as grid foldings.

2.3 Rigid Lens Measures for Change Detection

As mentioned above, we are interested in features for automatic change detection and visualization. Next we introduce three evident measures based on the hypothesis that relevant changes alter shape, size or appearance of structures. The first measure that we propose is the average deformation difference targeting changes in shape and size:

$$\begin{aligned} d_{\text {def}}= \frac{1}{|\mathcal {L}|}\sum _{x \in \mathcal {L}} \Vert y(x) - y_{\text {rigid}}(x)\Vert . \end{aligned}$$

It estimates the degree of local rigidity of y and therefore provides information about local morphological changes w.r.t. lengths and angles. The second measure aims at detection of changes in size. We define the average Jacobian as

$$\begin{aligned} d_{\text {jac}}= \frac{1}{|\mathcal {L}|}\sum _{x \in \mathcal {L}} \det \nabla y(x). \end{aligned}$$

Both measures \(d_{\text {def}}\) and \(d_{\text {jac}}\) are purely based on the computed deformation field only and do not take any image information into account, i.e., how the deformation affects the image appearance. To this end we propose the relative intensity difference quotient defined as

$$\begin{aligned} d_{\text {int}}= \frac{\sum _{x \in \mathcal {L}}|D(y_{\text {rigid}},x) - D(y,x)|}{\sum _{x\in \mathcal {L}} D(y_{\text {rigid}},x)} \end{aligned}$$

with difference image \(D(\phi ,x) := |R(x) - T(\phi (x))|\). Clearly, \(d_{\text {int}}\) only makes sense in a mono-modal setting as ours and aims on subtraction imaging. However, under the assumption that the deformation y computed by non-rigid registration leads to better alignment than its locally rigid approximation \(y_{\text {rigid}}\) we expect that \(D(y_{\text {rigid}},x) \ge D(y,x)\). Therefore values of \(d_{\text {int}}\) are expected in the range [0, 1] with \(d_{\text {int}}\approx 1\) if y leads to almost perfect alignment, such that R and T(y) are almost identical, and \(d_{\text {int}}\approx 0\) if y and \(y_{\text {rigid}}\) produce basically the same warped images, i.e., T(y) and \(T(y_{\text {rigid}})\) are almost identical. Note, that \(d_{\text {def}}\), \(d_{\text {jac}}\), \(d_{\text {int}}\) are local averages depending on the location and size of the lens region \(\mathcal {L}\equiv \mathcal {L}_r(x)\). Therefore, they can also be considered as point-wise measures at scale r, i.e., \(d_{\text {def}}\equiv d_{\text {def}}(r,x)\), \(d_{\text {jac}}\equiv d_{\text {jac}}(r,x)\) and \(d_{\text {int}}\equiv d_{\text {int}}(r,x)\).

3 Results

We demonstrate the rigid lens and evaluate our measures for change detection on CT follow-up thorax-abdomen scans of cancer patients. The CT data used for our experiments was collected from patients referred from the oncology department at the Radboud University Medical Center, Nijmegen, the Netherlands. In total we used, 1263 thorax-abdomen CT scans of 487 patients from different scanners and protocols with slice thickness varying from 1 mm to 2 mm. Furthermore, we used 2898 annotations of tumors made by the radiologists during reporting for quantitative evaluation of the rigid lens measures. We implemented the rigid lens as an interactive application in MeVisLab (http://www.mevislab.de), where the user hovers the lens over a reference image. The rigid deformation is instantaneously calculated from the given non-rigid deformation and the locally rigid warped template image is displayed inside the lens region. All computations are performed in real time on a state-of-the-art off-the-shelf PC. The application was used for the computations and visualization of the results described below.

3.1 A Motivating Example

Our first example illustrates the rigid lens concept for interactive viewing with a deformation that almost perfectly aligns the images and removes relevant local changes. Figure 2 shows an example of a lens region with a kidney tumor inside. The tumor in the non-rigid deformed template matches nearly perfect the tumor in the reference, such that we cannot observe changes in the difference image. In contrast, with the rigid lens we can see significant tumor growth in the rigidly deformed template. The difference between non-rigid registered and rigidly deformed template are quite high, which is also reflected by the rigid lens measures: \(d_{\text {def}}= 5.32\), \(d_{\text {int}}= 0.67\), and \(d_{\text {jac}}= 1.18\). The deformation in the lens region deviates 5.32 mm on average from the rigid deformation. This is also reflected by the Jacobian, that indicates volume growth of \(18\%\) in the lens region. Finally, our third measure \(d_{\text {int}}\) also takes a high value that indicates high intensity and appearance changes, respectively. Those high values are also visible in the intensity/deformation difference and Jacobian images shown in Fig. 2.

3.2 CT Follow-Up Registration

Now we look at thorax-abdomen CT follow-up scans, to demonstrate the utility of the proposed method. First we want to look on the visible effects and the qualitative information gain of the rigid lens in tumor regions. Afterwards we examine the quantitative benefit of the proposed algorithm in those regions.

Fig. 2.
figure 2

CT abdomen scan with a rigid lens on a kidney tumor region: the image top left shows the coronal view of the reference image with the rigid template lens region. In the second row from left to right the lens region of the reference, the registered and rigid deformed template are displayed. The images of the last row show the intensity differences between the registered and the rigidly approximated and the difference image between the reference and the registered/rigid deformed template. The image top middle shows deformation differences between the registered and the rigidly approximated template and the image on the right shows the Jacobian of the deformation.

Full size image
Fig. 3.
figure 3

Follow-up CT abdomen scan with a rigid lens on a liver tumor region: In the first row the sagittal view of the reference CT abdomen scan with the rigid template lens region and the lens region of the reference image are displayed. The rows from top to bottom show the rigidly approximated template region, the registered template region and intensity/deformation difference images between registered and rigid deformed template at the same position in the time follow-up template images.

Full size image

Qualitative Study. The image top left of Fig. 3 shows the sagittal view of a CT abdomen scan with the considered tumor region, where we want to analyze the shape and volume development of the tumor. The baseline scan served as reference for registration with three follow-up scans token two, six, and nine month later. In the lens region a large and a small tumor can be recognized. With the rigid lens we clearly recognize how the tumors grow, whereas we do not recognize such tumor changes in the registered template images as in the rigid deformed region: In the first column the registration matches the tumor regions of the template image quite well to the ones of the reference image, but in the images token later the corresponding regions match less perfect. These tumor changes are also visible in the intensity and deformation difference between the registered and the rigid deformed template: In the first image we do not recognize high intensity and deformation changes in the tumor region, so the transformation of this region is quite rigid. In the images on the right we see more intensity and deformation differences in the tumor region, since the difference between the rigid deformed and registered tumor get higher. These observations fit to the calculated rigid lens measures listed in Table 1: We measured high deformation changes, which increase through the time follow-up images. Furthermore the intensity quotients are always on a high level and the average Jacobians indicates an expansion of the volume vector field. In summary we assessed a growing and expansion of the tumor in the lens region by approximating the deformation field rigidly.

Table 1. Rigid lens measures to the corresponding follow-up data of Fig. 3. Each line shows the results of the corresponding template image.
Full size table
Fig. 4.
figure 4

Tumor regions with \(d_{\text {jac}}\ll 1\) and \(d_{\text {jac}}\gg 1\): In each column is from top to bottom the reference image, the original and rigid deformed template, and the absolute intensity differences between reference and original/rigid deformed template shown. We observed \(d_{\text {jac}}= 0.52\) in the first and \(d_{\text {jac}}= 2.57\) in the second column.

Full size image

Quantitative Study. In this experiment we do a quantitative evaluation of the rigid lens measures for change analysis of tumor regions. In total we considered 881 non-rigid follow-up registrations with the baseline image as reference and the corresponding follow-up images as templates. We used the tumor annotations (largest diameter and center) to define 1492 rigid lens regions and evaluated the average deformation difference, intensity difference and Jacobian in the region of interest. From in total 2898 annotations we only considered those with diameter \(\ge 2\)cm to avoid statistics based on very small lens regions only containing few pixels. Furthermore, to avoid duplicate measurements in our statistics we used the annotations from the baseline scan for all corresponding follow-up registrations. Only in cases when no annotations at baseline are available, then the ones from follow-up scans were used. The results of our study are summarized in Table 2. We observe a large range of deformation differences with values ranging from 0.05mm to 16.13mm. We also observed intensity quotients on a significant high level. In average volume change \(d_{\text {jac}}\) is near 1, but standard deviation is quite high \(>30\%\) indicating significant tumor growth or shrinkage. For better understanding we illustrate the meaning of \(d_{\text {jac}}\ll 1\) and \(d_{\text {jac}}\gg 1\) in Fig. 4.

Table 2. Rigid lens measures of the quantitative study: Mean and standard deviation of the measured values are listed for each tumor type.
Full size table

4 Conclusions

We proposed a simple approach for change assessment which is independent from any particular image registration method. We showed, that the rigid lens can be used to assess changes of volume, shape and appearance of structures. The benefits of the rigid lens are its interactive usage and its computationally cheap calculation in real time, yielding local rigid alignment without performing additional registration. Furthermore we introduced three measures for non-rigid local changes. We showed that the measures are generally able to indicate changes in shape, size and appearance. Finally we evaluated our tool for the assessment of tumor in follow-up CT scans and demonstrated the approach with a quantitative study. In future work, we aim to verify, that the rigid lens measures are sensitive to changes by correlating the results presented in Table 2 with ground-truth tumor growth. Furthermore, we aim to extent the approach for automatic change detection. Another interesting direction of research is the generalization of the rigid lens to comparison and change detection in multi-modal registration.