Similarity Registration Problems for 2D/3D Ultrasound Calibration

Abstract

We propose a minimal solution for the similarity registration (rigid pose and scale) between two sets of 3D lines, and also between a set of co-planar points and a set of 3D lines. The first problem is solved up to 8 discrete solutions with a minimum of 2 line-line correspondences, while the second is solved up to 4 discrete solutions using 4 point-line correspondences. We use these algorithms to perform the extrinsic calibration between a pose tracking sensor and a 2D/3D ultrasound (US) curvilinear probe using a tracked needle as calibration target. The needle is tracked as a 3D line, and is scanned by the ultrasound as either a 3D line (3D US) or as a 2D point (2D US). Since the scale factor that converts US scan units to metric coordinates is unknown, the calibration is formulated as a similarity registration problem. We present results with both synthetic and real data and show that the minimum solutions outperform the correspondent non-minimal linear formulations.

1 Introduction

Ultrasound (US) is a low-cost and real-time medical imaging technique in minimally invasive surgery and in percutaneous procedures. It observes information under the surface, so it is used to locate invisible details about vessels, nerves, or tumours. By tracking the pose of a 2D ultrasound probe (2D US) we can render 3D reconstructions from a collection of 2D slices [1], while a tracked 3D probe (3D US) is able to build large and detailed 3D models from a set of 3D scans [2]. Both 2D US and 3D US can also be used to guide other tracked medical instruments, such as biopsy needles [3], and fuse data with other imaging modalities such as endoscopes.

Freehand 3D ultrasound generally refers to the extrinsic calibration between a hand-held US probe and a pose tracking device. This calibration aims at determining the rigid transformation between the US scan and the tracked marker as well as the scale factor that converts the US scan to metric coordinates, i.e. a similarity transformation. This is usually achieved by scanning a known calibration object (phantom) immersed in either water or a tissue mimicking gel. Since the speed of sound in water is different than in tissue, sometimes an alcoholic solution is used to obtain a more realistic US scale. A multitude of calibration phantoms with different shapes have been proposed in the literature [4], including intersecting wires [5], a single plane [6, 7], a stylus [1, 8, 9], and 3D printed objects [10]. Although these methods focus on 2D US calibration, some extensions to 3D US using similar phantoms have been proposed as well [11, 12] (Fig. 1).

In this paper we focus on using a tracked needle as the calibration phantom. Our main motivation is towards assisted guidance and motion analysis in fetal interventions that require the extraction of in utero samples with a biopsy needle. It thus becomes a practical solution to use the same needle as a calibration object, avoiding the need to introduce new objects in the operating room and the additional burden of their sterilization. The tracked needle is detected by the pose tracking system as a 3D line, and it is scanned either as a line (3D US) or as a point (2D US). By scanning the needle under different poses, we formulate the 3D US calibration as the similarity registration between two sets of 3D lines and the 2D US calibration as the similarity registration between co-planar 3D points and 3D lines.

In this paper we propose a minimal solver to the similarity registration between two sets of 3D lines. We will also show that the registration between co-planar 3D points and 3D lines is a sub-problem of the same formulation and therefore the same minimal solver can be applied. Additionally, we show that this minimal solution can be easily generalised to the registration of any combination of plane, line, and point correspondences. We also present an alternative simplified minimal solver to the similarity registration between a set of co-planar points and a set of 3D lines. We apply the minimal solutions to the calibration of a 2D US and a 3D US with a pose tracking sensor and perform validation with both synthetic and real data.

Fig. 1.

(a) Scanning a tracked needle with a US probe; (b) a 2D US probe detects a cross section of the needle; (c) a 3D US detects a line segment.

2 Related Work

Freehand US calibration using a tracked linear target was proposed in [1]. However, this method is initialized with a non-minimal linear solution and is only meant for calibration of 2D US probes. Furthermore, it assumes that the US probe produces an anisotropic image, i.e. it has different scaling factors along the x and y axes of the image. An alternative method [13] extends this calibration procedure to 3D US and shows that assuming an isotropic model (single scale factor) produces better calibration results for curvilinear shaped probes. In this paper we assume that the US scans are isotropic. In some contexts, it is possible to assume that the scale factor is known, and the calibration problem becomes the Euclidean registration between the US probe and the phantom target. In the 2D US case this problem becomes equivalent to the extrinsic calibration between a camera and a laser rangefinder [14]. In the 3D US case, with the appropriate phantom (e.g. 3 known 3D points) the absolute pose of the probe can be recovered in each calibration scan, and thus it can be formulated as the standard hand-eye problem [15, 16]. In this paper, however, we consider that the scale is always unknown.

Estimating the similarity transformation (rigid pose and scale) between two coordinate frames gained recent attention due to its application in the registration of different Structure-from-Motion (SfM) sequences. If the same scene is recovered in two different monocular SfM runs, the scale of each reconstruction can be arbitrarily different. Therefore, to produce extended and more detailed 3D maps from independent SfM runs both the rigid pose and the scale must be recovered. If correspondences between SfM sequences are not available, one can use an extension of the ICP algorithm [17] to handle unknown scale [18]. If 2D-3D point correspondences are available, this is called the generalised pose and scale problem [19, 20], and is solved by extending the PnP formulation [21–23] to handle the alignment of image rays from multiple view points. A closely related contribution estimates a similarity transformation from pairwise point correspondences between two generalised cameras [24].

In the case of 3D US calibration, we are interested in the similarity registration between two sets of 3D lines. Different algorithms have been proposed to the euclidean registration between sets of lines [25, 26]. One possible approach to solve the similarity registration problem would be to first estimate the unknown scale factor independently, e.g. by computing the ratio of orthogonal distances between all pairs of lines in both sets, and then use any of the previously mentioned euclidean registration algorithms. We found that this approach is extremely unstable with noisy measurements and thus we focus on the joint estimation of all similarity parameters. Non-minimal linear algorithms and non-linear refinement methods have been proposed to solve the registration of two sets of 3D lines for different non-rigid configurations, including the similarity transformation [19, 27]. However, and to the best of our knowledge, a minimal closed-form solution for the similarity registration of two sets of lines have not been proposed in the literature.

The 2D US calibration problem is the similarity registration between a set of 3D lines and a set of co-planar points. This is a particular case of the pose and scale problem [19] when the 3D points are co-planar, and therefore this method could be adapted to solve this problem. However, the co-planarity of points introduces further simplifications, and as we will show in this paper, this problem can be minimally solved with a much more compact set of equations.

Fig. 2.

US probe models: (a) US emitted beams have a varying width. If we assume that the scanned region is focused, beams are approximated by a straight line. (b) linear 2D US (c) Curvilinear 2D US (d) curvilinear 3D US.

Our strategy to solve both registration problems is to convert them to an equivalent registration between a set of 3D points and a set of 3D planes. Although this strategy has been described in the context of euclidean registration [28], it is also valid for non-rigid registration.

Minimal solutions are a well established topic in computer vision literature [29–33]. In most cases they require solving a system of polynomial equations, which can be achieved using Grobner basis methods [30, 33, 34]. Although these methods provide a general framework to build numeric polynomial solvers, they require a certain amount of symbolic manipulation that often requires a case-by-case analysis. To address this issue an automatic generator of polynomial solvers have been proposed [32]. In this paper we develop minimum solutions using the action matrix method as presented in [34].

3 2D/3D US Model

US probes emit a set of acoustic beams that are partially reflected whenever they cross a medium interface with a change in acoustic impedance. The time response of the echo reflections enables the formation of a spatial grayscale map representing the different acoustic impedances within the US scanning field of view. Note that US beams have a varying width (Fig. 2(a)) which might induce undesired out of focus distortions. In an analogous way to most camera calibration models, we assume that the scanned region is always focused and thus each scene point reflects a beam along a single straight line.

US image formation depends on the probe construction. Linear 2D US probes (Fig. 2(b)) emit parallel beams and thus there are two scale factors involved: $s_{y}$ depends on the speed of sound in the propagation medium, while $s_{x}$ is a fixed parameter that depends on the physical distance between beam emitters. These probes usually operate with high frequency acoustic signals (4–16 MHz) and are used for short range scans (e.g. musculoskeletal imaging). The calibration of these probes cannot be represented with a similarity transformation and we discard it from further analysis in this paper. Curvilinear 2D US probes (Fig. 2(c)) emit beams in radial directions that intersect in a single point, forming a planar bundle of lines. In this case, the speed of sound in the propagation medium affects the scan scale isotropically. The curvilinear 3D US (Fig. 2(d)) is a generalization of the curvilinear 2D US, emitting a 3D bundle of beams. Curvilinear probes usually operate with lower frequency signals (2–8 MHz) and are more suitable to long range scans (e.g. obstetrics, cardiac imaging).

Fig. 3.

3D US calibration problem. The similarity transformation $\mathsf {A}$ maps points $\mathbf {X}_{i}$ in the 3D US volume (red) to points $\mathbf {P}_{i}$ in the marker reference frame (green). $\mathsf {A}$ can be decomposed as a uniform scaling transformation followed by a rigid transformation. (Color figure online)

4 Problem Statement

Consider a hand-held curvilinear 3D US probe whose pose is tracked by a rigidly attached marker (Fig. 3). In each frame the tracking system determines the transformation $\mathsf {T}_{\mathsf {M} \rightarrow \mathsf {O}}$ from the marker coordinate system ($\mathsf {M}$) to a fixed frame $\mathsf {O}$. The freehand 3D US calibration consists in determining the unknown similarity transformation $\mathsf {A}$ that maps 3D points $\mathbf {X}_{i}$ in the US volume to 3D points $\mathbf {P}_{i}$ represented in $\mathsf {M}$.

$$\begin{aligned} \mathbf {P}_{i} = \mathsf {A} \mathbf {X}_{i} \end{aligned}$$

(1)

The similarity $\mathsf {A}$ is defined by a rotation $\mathsf {R}$, a translation $\mathbf {t}$ and a scale factor s that converts the 3D US volume to metric coordinates, and is represented as

$$\begin{aligned} \mathsf {A} = \begin{pmatrix} \mathsf {S} &{} \mathbf {t} \\ 0 &{} 1 \end{pmatrix} \end{aligned}$$

(2)

where $\mathsf {S} = s \mathsf {R}$ is a scaled rotation matrix such that

$$\begin{aligned} {\mathsf {S}}^\mathsf {T} \mathsf {S} = \mathsf {S} {\mathsf {S}}^\mathsf {T} = \begin{pmatrix} s^{2} &{} 0 &{} 0 \\ 0 &{} s^{2} &{} 0 \\ 0 &{} 0 &{} s^{2} \end{pmatrix} \end{aligned}$$

(3)

The calibration procedure consists in capturing a tracked needle in the 3D US volume under different poses. The needle is previously calibrated by determining its two endpoints in the reference frame $\mathsf {M}$ and then it is represented in each acquisition as a 3D line $\mathbf {L}_{i}$. The needle is also detected as a 3D line $\mathbf {B}_{i}$ in the 3D US volume. The calibration problem is thus formulated as the 3D similarity registration between two sets of lines (Fig. 4(a)).

Fig. 4.

(a) The 3D US calibration is formulated as the similarity registration between lines $\mathbf {B}_{i}$ and lines $\mathbf {L}_{i}$; (b) Each line $\mathbf {L}_{i}$ can be re-defined as two intersecting planes $\mathbf {\Pi }_{i}$, $\mathbf {\Pi }^{*}_{i}$, while each line $\mathbf {B}_{i}$ can be redefined as two points $\mathbf {X}_{i}$, $\mathbf {X}^{*}_{i}$.

5 3D US Calibration Solution

In this section we derive a minimal solution for the calibration of a 3D US probe. We start by re-stating it as the similarity registration between 3D points and 3D planes, and then derive a linear and a minimal solution for this problem. The calibration of a 2D US is presented in Sect. 6 as a particular case of the 3D US problem.

5.1 3D US Calibration as Point-Plane Registration

Ramalingam et al. showed that any 3D registration problem involving 3D planes, lines and/or points can be re-stated as the registration between 3D planes and 3D points [28]. In our calibration problem this can be achieved by defining each needle line $\mathbf {L}_{i}$ as two intersecting planes $\mathbf {\Pi }_{i}$, $\mathbf {\Pi }^{*}_{i}$ and each line $\mathbf {B}_{i}$ as two points $\mathbf {X}_{i}$, $\mathbf {X}^{*}_{i}$ (Fig. 4(b)).

Given that both $\mathbf {P}_{i} = \mathsf {A} \mathbf {X}_{i}$ and $\mathbf {P}^{*}_{i} = \mathsf {A} \mathbf {X}^{*}_{i}$ are contained in planes $\mathbf {\Pi }_{i}$ and $\mathbf {\Pi }^{*}_{i}$, each line-line correspondence ($\mathbf {L}_{i}$,$\mathbf {B}_{i}$) puts 4 linear constraints on $\mathsf {A}$

$$\begin{aligned} {\mathbf {\Pi }_{i}}^\mathsf {T}{\mathsf {A}} \mathbf {X}_{i} = 0 \end{aligned}$$

(4)

$$\begin{aligned} {\mathbf {\Pi }^{*}_{i}}^\mathsf {T}{\mathsf {A}} \mathbf {X}_{i} = 0 \end{aligned}$$

(5)

$$\begin{aligned} {\mathbf {\Pi }_{i}}^\mathsf {T}{\mathsf {A}} \mathbf {X}^{*}_{i} = 0 \end{aligned}$$

(6)

$$\begin{aligned} {\mathbf {\Pi }^{*}_{i}}^\mathsf {T}{\mathsf {A}} \mathbf {X}^{*}_{i} = 0 \end{aligned}$$

(7)

Note that the same reasoning can be applied to any combination of plane, point, and line correspondences (planes are defined by 3 points, and points are defined as the intersection of 3 planes), and thus the remainder of this section equally applies to these problems as well.

5.2 Linear Solution

The similarity matrix $\mathsf {A}$ has 13 linear parameters and for N line-line correspondences we can stack instances of the Eqs. 4, 5, 6 and 7) to form a linear system with 4N equations and 13 unknowns. This linear system can be solved with SVD decomposition using at least 3 correspondences, determining $\mathsf {A}$ up to a scale factor. The correct scale of $\mathsf {A}$ can be recovered by setting the homogeneous parameter to 1. Note, however, that with noisy line measurements Eq. 3 is not satisfied and thus the linear solution for $\mathsf {A}$ is generally not a similarity. The linear estimation can be projected to a similarity using QR decomposition of matrix $\mathsf {S}$ and forcing its upper triangular component to be a scaled identity matrix (using the mean of its diagonal elements as scale s)

$$\begin{aligned} \mathsf {S} = s\mathsf {R} = \mathsf {R}\, \begin{pmatrix} s &{} 0 &{} 0 \\ 0 &{} s &{} 0 \\ 0 &{} 0 &{} s \\ \end{pmatrix} \end{aligned}$$

(8)

5.3 Minimal Solution

Equation 3 puts 5 quadratic constraints on matrix $\mathsf {A}$ and therefore only 7 linear constraints are required to compute its 13 parameters. This can be achieved with a minimum of 2 line-line correspondences. Note that with 2 correspondences we have 8 linear constraints. To solve the problem minimally we should either discard one of the linear equations or partially solve the complete linear system, leaving 6 up to scale unknowns undetermined. We found the latter option to be numerically more stable. The linear system with 7 equations and 13 unknowns is partially solved using SVD decomposition, generating a 6D solution subspace for $\mathsf {A}$

$$\begin{aligned} \mathsf {A} = a \mathsf {A}_{a} + b \mathsf {A}_{b} + c \mathsf {A}_{c} + d \mathsf {A}_{d} + e \mathsf {A}_{e} + f \mathsf {A}_{f} \end{aligned}$$

(9)

where a, b, c, d, e, f are the remaining 6 unknowns.

Equation 3 can be written as the following system of 10 quadratic equations

$$\begin{aligned} \begin{aligned} {\mathbf {c}_{1}}^\mathsf {T} \mathbf {c}_{1} - {\mathbf {c}_{2}}^\mathsf {T} \mathbf {c}_{2}&= 0 \\ {\mathbf {c}_{1}}^\mathsf {T} \mathbf {c}_{1} - {\mathbf {c}_{3}}^\mathsf {T} \mathbf {c}_{3}&= 0 \\ {\mathbf {c}_{1}}^\mathsf {T} \mathbf {c}_{2}&= 0 \\ {\mathbf {c}_{1}}^\mathsf {T} \mathbf {c}_{3}&= 0 \\ {\mathbf {c}_{2}}^\mathsf {T} \mathbf {c}_{3}&= 0 \end{aligned} \qquad \qquad \qquad \begin{aligned} {\mathbf {r}_{1}}^\mathsf {T} \mathbf {r}_{1} - {\mathbf {r}_{2}}^\mathsf {T} \mathbf {r}_{2}&= 0 \\ {\mathbf {r}_{1}}^\mathsf {T} \mathbf {r}_{1} - {\mathbf {r}_{3}}^\mathsf {T} \mathbf {r}_{3}&= 0 \\ {\mathbf {r}_{1}}^\mathsf {T} \mathbf {r}_{2}&= 0 \\ {\mathbf {r}_{1}}^\mathsf {T} \mathbf {r}_{3}&= 0 \\ {\mathbf {r}_{2}}^\mathsf {T} \mathbf {r}_{3}&= 0 \end{aligned} \end{aligned}$$

(10)

where $\mathbf {c}_{i}$ is the ith column of $\mathsf {S}$ and $\mathbf {r}_{i}$ is the ith row of $\mathsf {S}$. Substituting Eq. 9 into Eq. 10 generates a system of 10 quadratic equations in the 6 unknowns a, b, c, d, e, f. Note that this polynomial system is the same solved in [19] for the Generalised Pose and Scale Problem.

Fig. 5.

(a) The 2D US calibration is formulated as the similarity registration between co-planar points $\mathbf {P}_{i}$ and lines $\mathbf {L}_{i}$; (b) each line $\mathbf {L}_{i}$ can be re-defined as two intersecting planes $\mathbf {\Pi }_{i}$, $\mathbf {\Pi }^{*}_{i}$.

This polynomial system is solved with the action matrix method [34]. Since the quadratic constraints determine $\mathsf {A}$ up to scale we set $f = 1$. We expand the polynomial system by multiplying all equations by a, b, c, d and form a cubic system with 47 linearly independent equations and 55 monomials. Using LU decomposition, we reduce the system to 5 equations in 13 monomials

$$\begin{aligned} \begin{pmatrix}\mathsf {C}_{5 \times 5}&\mathsf {B}_{5 \times 8} \end{pmatrix} \begin{pmatrix} \mathbf {m}_{C} \\ \mathbf {m}_{B}\end{pmatrix} = 0 \end{aligned}$$

(11)

with

$$\begin{aligned} \mathbf {m}_{C}&= {\begin{pmatrix} b^3&ab^2&be&bd&bc \end{pmatrix}}^\mathsf {T} \end{aligned}$$

(12)

$$\begin{aligned} \mathbf {m}_{B}&= {\begin{pmatrix} b^2&ab&e&d&c&b&a&1\end{pmatrix}}^\mathsf {T} \end{aligned}$$

(13)

When a polynomial system is presented in this format, it can be solved with the action matrix method if matrix $\mathsf {C}_{5 \times 5}$ is invertible and also if there is a monomial w such that $w \mathbf {m}_{B}$ is a linear combination of $\mathbf {m}_{B}$. In our calibration problem $\mathsf {C}_{5 \times 5}$ is generally invertible, and for $w = b$ we can build a $8 \times 8$ matrix $\mathsf {M}$ such that

$$\begin{aligned} \mathsf {M} \mathbf {m}_{B} = b \mathbf {m}_{B} \end{aligned}$$

(14)

The 8 solutions to $\mathbf {m}_{B}$ that verify this constraint are the eigen vectors of $\mathsf {M}$, from which we can extract 8 solutions for a, b, c, d, e and recover 8 solutions for $\mathsf {A}$ using Eq. 9. The correct scale of $\mathsf {A}$ is recovered in the same way as explained in Sect. 5.2.

6 2D US Calibration Solution

If we consider the same calibration problem with a curvilinear 2D US probe instead, each needle acquisition is detected as a single point $\mathbf {X}_{i}$ that belongs to the US scanning plane. For the sake of continuity with the previous section, we still treat the image coordinates $\mathbf {X}_{i}$ of the 2D US as 3D co-planar points, for an arbitrarily fixed scanning plane $\mathbf {\Delta }$. Note that calibrating a curvilinear 2D US aims at determining the same 7 parameters as in the 3D US case. Therefore the calibration problem becomes the 3D similarity registration between a set of co-planar points $\mathbf {X}_{i}$ and a set of lines $\mathbf {L}_{i}$ Fig. (5(a)).

Each point-line correspondence puts 2 linear constraints on matrix $\mathsf {A}$ (Eqs. 4 and 5), and therefore this problem can be minimally solved using 4 point-line correspondences. The same minimal solution described in Sect. 5.3 can be used in this case, since the co-planarity of points $\mathbf {X}_{i}$ is not a degenerate configuration. We observed that some particular choices for the scanning plane $\mathbf {\Delta }$ (e.g. $z=0$) result in matrix $\mathsf {C}_{5 \times 5}$ being singular and thus the polynomial system becomes numerically unstable. We found out through simulation that defining $\mathbf {\Delta }$ as the plane $z=k$, with $k>0$ generally produces an invertible matrix $\mathsf {C}_{5 \times 5}$ and the polynomial system is solvable.

On the other hand, the linear solution described in Sect. 5.2 will not solve the 2D US problem, as the system will always be rank deficient. This can only be achieved with the additional elimination of parameters in the linear equations. These simplifications also lead to an alternative minimal solution for the 2D US case. Both methods are described in the remainder of this section.

6.1 Linear Solution

If we define the scanning plane $\mathbf {\Delta }$ as $z=0$, the 2D US points have the format $\mathbf {X}_{i} = {\begin{pmatrix} x_{i}&y_{i}&0&1 \end{pmatrix}}^\mathsf {T}$ and the linear equations do not put any constraints on the third column of $\mathsf {S}$. The linear equations for each acquisition become

$$\begin{aligned} {\mathbf {\Pi }_{i}}^\mathsf {T} \bar{\mathsf {A}} {\begin{pmatrix} x_{i}&y_{i}&1 \end{pmatrix}}^\mathsf {T}&= 0 \end{aligned}$$

(15)

$$\begin{aligned} {\mathbf {\Pi }^{*}_{i}}^\mathsf {T} \bar{\mathsf {A}} {\begin{pmatrix} x_{i}&y_{i}&1 \end{pmatrix}}^\mathsf {T}&= 0 \end{aligned}$$

(16)

with

$$\begin{aligned} \bar{\mathsf {A}}&= \begin{pmatrix} \bar{\mathsf {S}} &{} \mathbf {t} \\ 0 &{} 1 \end{pmatrix}\end{aligned}$$

(17)

$$\begin{aligned} \bar{\mathsf {S}}&= \begin{pmatrix} \mathbf {c}_{1}&\mathbf {c}_{2} \end{pmatrix} \end{aligned}$$

(18)

where $\mathbf {c}_{1}$ and $\mathbf {c}_{2}$ are the first two columns of $\mathsf {S}$. The linear system is thus reduced to 10 unknown parameters and can be solved with a minimum of 5 point-line correspondences. Note that analogously to the 3D US case (Eq. 3), $\bar{\mathsf {A}}$ must verify the following constraint

$$\begin{aligned} \bar{\mathsf {S}} {\bar{\mathsf {S}}}^\mathsf {T} = \begin{pmatrix} s^{2} &{} 0 \\ 0 &{} s^{2} \end{pmatrix} \end{aligned}$$

(19)

and with noisy measurements the linear solution must be forced to this format using its QR decomposition

$$\begin{aligned} \bar{\mathsf {S}} = \mathsf {R}\, \begin{pmatrix} s &{} 0 \\ 0 &{} s \\ 0 &{} 0 \end{pmatrix} \end{aligned}$$

(20)

The third column of $\mathsf {S}$ can then be extracted by multiplying the third column of rotation $\mathsf {R}$ by s.

6.2 Alternative Minimal Solution (2D US Only)

This problem can be minimally solved with 7 linear constraints (4 point-line correspondences). Since in this case there are only 10 linear parameters, we can generate a 3D linear solution subspace

$$\begin{aligned} \bar{\mathsf {A}} = a \bar{\mathsf {A}}_{a} + b \bar{\mathsf {A}}_{b} + c \bar{\mathsf {A}}_{c} \end{aligned}$$

(21)

Equation 19 is re-written as the following system

$$\begin{aligned} \begin{aligned} {\mathbf {c}_{1}}^\mathsf {T} \mathbf {c}_{1} - {\mathbf {c}_{2}}^\mathsf {T} \mathbf {c}_{2}&= 0 \\ {\mathbf {c}_{1}}^\mathsf {T} \mathbf {c}_{2}&= 0 \end{aligned} \end{aligned}$$

(22)

Substituting Eq. 21 into Eq. 22 we generate a system of 2 quadratic homogeneous equations in the 3 unknowns a, b, c. Using the same procedure from Sect. 5.3 we use monomial multiplication and LU decomposition to re-write this system as

$$\begin{aligned} \begin{pmatrix} \mathsf {C}_{2 \times 2}&\mathsf {B}_{2 \times 4} \end{pmatrix} \begin{pmatrix} \mathbf {m}_{C} \\ \mathbf {m}_{B} \end{pmatrix} = 0 \end{aligned}$$

(23)

with

$$\begin{aligned} \mathbf {m}_{C}&= {\begin{pmatrix} ab^{2}&b^{2} \end{pmatrix}}^\mathsf {T} \end{aligned}$$

(24)

$$\begin{aligned} \mathbf {m}_{B}&= {\begin{pmatrix} ab&b&a&1 \end{pmatrix}}^\mathsf {T} \end{aligned}$$

(25)

We solve this system using eigen decomposition of the action matrix, yielding up to 4 solutions.

7 Degenerate Cases

The degenerate configurations for both 3D US and 2D US calibration are closely related to the ones described for the pose and scale problem [19]. If the needle is moved without rotation (lines $\mathbf {L}_{i}$ are parallel) there is an ambiguity in translation. This implies that fixing the needle and scanning with the US probe in different positions is a degenerate case, however, the inverse scenario of fixing the US probe while moving the needle is generally not a degenerate case. If the needle motion is a pure rotation around itself (lines $\mathbf {L}_{i}$ intersect in a single point) there is an ambiguity in scale. This is analogous to pose estimation with monocular pinhole cameras. If lines $\mathbf {L}_{i}$ are co-planar, the point detections $\mathbf {X}_{i}$ of a 2D US are co-linear and there is a rotation ambiguity around the axis defined by these points. This, however, is not generally a degenerate case in 3D US calibration unless only 2 line correspondences are available, since it falls under either one of the two previously mentioned cases. Therefore, the similarity between two sets of co-planar lines can only be estimated from a minimum of 3 correspondences.

Fig. 6.

The needle detections in 3D are obtained by sampling 2D slices: (a) simulated set-up with a fixed 3D US scanner and random needle poses in green; (b) 3D US line acquisition; 2D US point acquisition. (Color figure online)

8 Iterative Refinement

The closed-form solutions can be refined with Levenberg-Marquardt iterative optimization [35], however, there is no consensus on the most appropriate residue metric for 3D line registration [27]. In all our experiments we perform iterative refinement by minimizing the euclidean orthogonal distance between the 3D lines $\mathbf {L}_{i}$ and the projected 3D points from the US image $\mathbf {P}_{i} = \mathsf {A} \mathbf {X}_{i}$. The refined solution is parametrised by the translation $\mathbf {t}$, 3 rotation parameters ($\mathbf {R}$ is represented as a unit norm quaternion), and the scale factor s. For the 3D US the optimization problem is

$$\begin{aligned} \min _{\mathsf {R}, \mathbf {t}, s} \sum _{i = 1}^{N} d(\mathbf {L}_{i},\mathbf {P}_{i})^{2} + d(\mathbf {L}_{i},\mathbf {P}^{*}_{i})^{2} \end{aligned}$$

(26)

where $d(\mathbf {L}_{i},\mathbf {P}_{i})$ represents the euclidean distance between line $\mathbf {L}_{i}$ and point $\mathbf {P}_{i}$. For the 2D US problem the last term of the minimization is ignored.

9 Experimental Results

We test the calibration algorithms both in simulation and with real data. For the 3D US calibration we test the linear solution from Sect. 5.2 (3line3D) and the minimum solution from Sect. 5.3 (2line3D), while for the 2D US we test the linear solution from Sect. 6.1 (5point2D), the general minimal solution from Sect. 5.3 (2line3D), and the simplified minimum solution from Sect. 6.2 (4point2D). All algorithms are tested within a RANSAC framework [36] with outlier threshold of 5 mm, followed by iterative refinement.

With both synthetic and real data, the 3D US lines $\mathbf {B}_{i}$ are obtained by sampling points from 2D slices with different angles (Fig. 6). This is is a practical solution since we can directly define the points $\mathbf {X}_{i}$ and $\mathbf {X}^{*}_{i}$ to input in Eqs. 4, 5, 6 and 7. The needle tracking measurements (lines $\mathbf {L}_{i}$) are converted to two intersecting planes $\mathbf {\Pi }_{i}$ and $\mathbf {\Pi }^{*}_{i}$ such that $\mathbf {\Pi }_{i}$ intersects both $\mathbf {L}_{i}$ and the origin of the US attached marker reference frame $\mathsf {M}$, and $\mathbf {\Pi }_{i}^{*}$ is orthogonal to $\mathbf {\Pi }{i}$. Note that this approach degenerates when the needle is aligned with the origin of $\mathsf {M}$ and therefore this should be taken into account when positioning the needle during calibration.

Fig. 7.

2D US error distributions with synthetic data

Fig. 8.

3D US error distributions with synthetic data

All plots in this section are represented with Matlab boxplot function: the central mark is the median, the box limits are the 25th and 75th percentiles, the whiskers are the maximum and minimum inliers, and individual crosses are outliers.

9.1 Simulation

We simulate a 2D/3D US probe with a scale factor $s=0.24$ in a fixed position. 50 line segments with 400 mm are generated at random poses within the field of view of the US. Gaussian noise is added to the US points along the 2D slices ($\sigma = 1$ pixel) and also to the extreme points of the line segments $\mathbf {L}_{i}$ ($\sigma = 1$mm) to simulate tracking error. In each trial, we calibrate the US by sampling N random line segments. N varies between 3 and 10 for the 3D US case, and between 5 and 10 for the 2D US case. For each value N we perform 100 trials. The calibration results are compared against of rotation ($\mathsf {R}_{GT}$), translation ($\mathbf {t}_{GT}$), and scale factor ($s_{GT}$) and are presented in Figs. 7 and 8. The rotation error is measured as the angle displacement of the residual rotation ${\mathsf {R}}^\mathsf {T} \mathsf {R}_{GT}$, the translation error as $||\mathbf {t}_{GT} - \mathbf {t}||$ and the scale error as $|s_{GT} - s|$. As expected, the minimal solutions perform better than the linear solutions with a low number of input acquisitions and converge to the same result as the number of acquisitions grow. Also not surprisingly, 3D US performs better that 2D US for the same number of acquisitions, given that it has twice as much linear constraints. In the case of 2D US, the two alternative minimum solutions have similar performance.

Fig. 9.

(a) Calibration set-up with a tracked 3D US probe and a tracked needle; (b) validation using a cross wire pattern that defines a known 3D point in the tracker reference frame.

9.2 Real Data

Our calibration method is tested using the set-up displayed in Fig. 9(a) that includes a GE Voluson E10 machine with a eM6C probe (3D US) and a 333 mm length metal needle. Both instruments are tracked by the infrared camera system Optitrack V120 Trio. Experiments were conducted in a container filled with water at room temperature. We use the same probe for both for 2D US and 3D US data acquisition. For the 2D US we just choose a single 2D slice from the 3D US volume at a specified angle. The needle is manually segmented as a point in each 2D slice. Unlike in the simulation experiment, in this calibration procedure both the needle and the 3D US probe are moved between acquisitions.

To validate the calibration accuracy we use an x-shaped wire phantom Fig. 9(b) whose point intersection can be measured as a single point in the US scan. We use this phantom to measure the projection reconstruction accuracy (PRA) of our calibration results, i.e., the difference in mm between the intersection point $\mathsf {A} \mathbf {X}$ according to the calibrated US measurement and the same point $\mathbf {P}$ measured by the tip of the tracked needle. We performed 10 acquisitions of the wire phantom in order to cover different regions of the US scan. Figures 10(a) and 10(b) display the distribution of PRA results for all trials. Each distribution contains 200 error measurements (20 trials $\times$ 10 phantom scans). In the 2D US results we only display the results for one of the minimal solutions, since as we’ve seen in simulation the results from both approaches are very similar.

Fig. 10.

Validation results: (a) projection reconstruction accuracy (PRA) with 3D US; (b) PRA with 2D US (c) sample registration result with 3D US projected lines in red and needle tracking measurements in green (d) sample registration result with 2D US projected points in red and needle tracking measurements in green (Color figure online)

The US calibration converges to a solution with an error between 2 and 3 mm within a total scanning radius of 120 mm. Since in each trial we select random needle poses, close to degenerate configurations can be chosen and result in outlier results. This can be avoided in practice by scanning the needle in a spread region of the US volume, and by exploring all 6 degrees of freedom while moving the needle. Overall, the difference in accuracy between linear and minimal solutions is even more pronounced than in simulation.

10 Conclusions

We propose a minimum solution to the similarity registration between two sets of 3D lines, and between co-planar points and 3D lines. These solutions are tested to calibrate a US probe using a tracked line target with both 3D and 2D data. This is useful in medical imaging to guide a biopsy needle during US based interventions. The method can be easily be extended to additional US calibration problems using other types of phantoms, e.g. scanning single plane target leads to the similarity registration between co-planar lines and 3D planes (2D US) or between two sets of 3D planes (3D US). In other computer vision domains this algorithm can potentially be used as an extension of the pose and scale problem to the alignment of line-based and/or plane-based SfM sequences.