Nasal Mesh Unfolding – An Approach to Obtaining 2-D Skin Templates from 3-D Nose Models

Abstract

Nasal reconstruction requires a 2-D template representing the skin area to be taken from the donor site of the patient. We propose a new framework for template design, called nasal mesh unfolding, to obtain 2-D skin templates from 3-D nose models. The proposed nasal mesh unfolding framework takes as input a target digital nose model represented by a 3-D triangle mesh and unfolds the nasal mesh under structure constraints using semidefinite programming. The solution of the unfolding problem is in the form of a Gram matrix from which the 2-D representation of the 3-D model, or embedding, is extracted. The embedding defines a digital template representing the skin requirement for nasal reconstruction, which can in turn be used to produce a physical 2-D template to apply on the donor site for guiding skin incision. Experiments on synthetic data demonstrate the effectiveness of the proposed unfolding approach, and results on real data show the feasibility of generating physical 2-D skin templates from 3-D nose meshes. The proposed approach efficiently converts 3-D nose models to digital 2-D skin templates for fast easy and accurate preparation of physical templates and can be useful for other plastic surgery tasks.

1 Introduction

Nasal reconstruction aims to restore aesthetics while preserving function. It is a challenging task due to the complexity of nasal anatomy. The nasal cover is of varying thickness and has contours with intersecting concavities and convexities [1]. Its reconstruction requires defining the target nose dimensions and estimating the corresponding skin requirement.

Sultan and Byrne worked with certified anaplastologists and proposed to define target nose dimensions physically by sculpting a wax model according to patients’ expectations [2, 3]. Hierl et al. [4] defined a 3-D nose shape digitally by an accepted virtual surgery result. When a physical model is available, its shape can be transformed to a 2-D skin template by molding a piece of foil over the model and then by flattening the foil back [3]. Aquaplast can be employed similarly instead of foil, and the resulting 3-D surface template can be converted to a 2-D skin template by releasing cuts [5]. But the precision loss caused by this molding and flattening process still exists and is difficult to quantify. Digital models provide an opportunity to quantify the template accuracy, but at present they do not bring much more information than nose photos, in which case experienced operators are demanded to delineate 2-D skin templates. Optionally we can produce 3-D models, e.g., by 3-D printing, and fall back to the molding and flattening process for 2-D template production.

In this work, we introduce a framework called nasal mesh unfolding to bridge the gap between a digital 3-D nose model and a 2-D skin template. We assume that the nose model is a 3-D surface represented by a triangle mesh (obtained for example by scanning a healthy nose) and we focus on unfolding this nasal mesh with the least possible distortion. We propose to unfold the 3-D mesh using a nonlinear dimensionality-reduction technique (see [6] for an overview), namely, semidefinite embedding [7], which positions mesh vertices as far apart as possible under mesh structure constraints. The resulting 2-D embedding is a digital template that can be traced on paper or foil for outlining the skin requirement on the donor site.

2 Materials and Methods

2.1 3-D Nasal Mesh Representation

The nose model is a triangle mesh represented by an ordered list of 3-D vertices $\mathcal {V} := (\varvec{v}_1,\ldots , \varvec{v}_n)$ and an $n \times n$ adjacency matrix $\varvec{A} := [a_{ij}]$ defining the connected vertex pairs, i.e., $a_{ij}=1$ indicates that there is an edge between vertices $\varvec{v}_i$ and $\varvec{v}_j$, and $a_{ij}=0$ otherwise. Any three vertices $\varvec{v}_i$, $\varvec{v}_j$, and $\varvec{v}_k$ form a mesh triangle if $a_{ij} = a_{ik} = a_{jk} = 1$. We denote the set of mesh triangles by $\mathcal {T}$. Given a vertex $\varvec{v}_i \in \mathcal {V}$, we let $\mathcal {V}(\varvec{v}_i)$ be the set of vertices connected to $\varvec{v}_i$ and $\mathcal {T}(\varvec{v}_i)$ be the set of mesh triangles containing $\varvec{v}_i$, i.e., $\mathcal {V}(\varvec{v}_i) := \{ \varvec{v}_j \in \mathcal {V} \,|\, a_{ij} = 1 \}$ and $\mathcal {T}(\varvec{v}_i) := \{\varvec{\tau } \in \mathcal {T} \,|\, \varvec{v}_i \in \varvec{\tau } \}$. We call $\mathcal {T}(\varvec{v}_i)$ the mesh patch centered on $\varvec{v}_i$, and we denote by $\overline{A}(\varvec{v}_i)$ the average area of the triangles in $\mathcal {T}(\varvec{v}_i)$.

In addition to the adjacency matrix $\varvec{A}$ representing the triangle mesh, we define an $n \times n$ symmetric binary matrix $\varvec{B} := [b_{ij}]$ representing constraints on mesh patches with large areas. Formally, $b_{ij}=1$ if and only if the following conditions hold: $\varvec{v}_j \not \in \mathcal {V}(\varvec{v}_i) \cup \{\varvec{v}_i\}$ and there is a vertice $\varvec{v}_k$ such that $\{\varvec{v}_i,\varvec{v}_j\} \subset \mathcal {V}(\varvec{v}_k)$ and $\overline{A}(\varvec{v}_k) \geqslant \alpha$, where $\alpha$ is a threshold set to the sum of the average and standard deviation of the mesh-triangle areas in the input 3-D mesh in our experiments. The construction of the matrix $\varvec{B}$ is illustrated in Fig. 1 for mesh patches centered on vertices of degree 4 and 5. The purpose of the matrix $\varvec{B}$ is to preserve large-area mesh patches during the unfolding process.

Fig. 1.

Illustration of the constraints on large-area mesh patches centered on vertices of degree 4 and 5. The solid line segments are the edges of the triangle mesh represented by the adjacency matrix $\varvec{A}$. The dashed line segments are the mesh-patch constraints added to the triangle mesh if $\overline{A}(\varvec{v}_1) \geqslant \alpha$, where $\overline{A}(\varvec{v}_1)$ is the average area of all the mesh triangles containing $\varvec{v}_1$.

2.2 The Mesh Unfolding Problem

Given the triangle mesh $(\mathcal {V},\varvec{A})$ modeling the desired nose shape, the unfolding process aims to embed the 3-D vertices $\varvec{v}_1,\ldots , \varvec{v}_n$ in the Euclidean plane while preserving the lengths of the edges encoded in the adjacency matrix $\varvec{A}$ and in the mesh-patch constraint matrix $\varvec{B}$. The correspondence between $\mathcal {V}$ and its 2-D embedding $(\varvec{x}_1,\ldots ,\varvec{x}_n)$ is obtained by maximizing the sum of the squared distances between the pairs of 2-D vertices under the edge-length isometry constraints indexed by the binary matrix $\varvec{A}+\varvec{B}$. Let $\Vert \cdot \Vert$ denote the standard Euclidean norm, and let $\mathcal {I}$ be the set of index pairs $\{i,j\}$ such that $\{\varvec{v}_i,\varvec{v}_j\}$ is an edge or a mesh-patch constraint, i.e.,

$$\begin{aligned} \mathcal {I} \,:=\, \big \{ \{i,j\} \subset \{1,\ldots ,n\} \,|\, a_{ij} + b_{ij} = 1 \big \}. \end{aligned}$$

(1)

The unfolding problem is the following:

$$\begin{aligned} \begin{array}{rl} \text {maximize} &{} \!\!{\displaystyle \sum _{i,j=1}^n \Vert \varvec{x}_i-\varvec{x}_j \Vert ^2 } \\ \text {subject to} &{} \!\! {\left\{ \begin{array}{ll} \, \forall \{i,j\} \in \mathcal {I}, \;\Vert \varvec{x}_i-\varvec{x}_j \Vert = \Vert \varvec{v}_i-\varvec{v}_j \Vert \!\! \\ \, {\displaystyle \sum _{i=1}^n \varvec{x}_i \,=\, \varvec{0}}, \end{array}\right. } \end{array} \end{aligned}$$

(2)

where the additional constraint $\sum _i \varvec{x}_i = \varvec{0}$ is intended to remove the translational degree of freedom.

2.3 Unfolding by Semidefinite Programming

Given a set of points $\mathcal {X}:=(\varvec{x}_1,\ldots ,\varvec{x}_n)$ in the Euclidean plane, we let $\varvec{G}:=[g_{ij}]$ be the Gram matrix of $\mathcal {X}$; i.e., for every $(i,j)\in \{1,\ldots ,n\}^2$, $g_{ij}$ is the dot product of $\varvec{x}_i$ and $\varvec{x}_j$, denoted by $\varvec{x}_i \cdot \varvec{x}_j$. Since $\Vert \varvec{x}_i-\varvec{x}_j\Vert ^2=g_{ii}-2g_{ij}+g_{jj}$ and $\Vert \sum _i\varvec{x}_i\Vert ^2=\sum _{i,j}\varvec{x}_i\cdot \varvec{x}_j$, the constraints in (2) are equivalent to

$$\begin{aligned} g_{ii} - 2g_{ij} + g_{jj} \,=\, \Vert \varvec{v}_i- \varvec{v}_j \Vert ^2 \end{aligned}$$

(3)

and

$$\begin{aligned} \sum _{i,j=1}^n g_{ij} = 0. \end{aligned}$$

(4)

Besides, the objective function of the unfolding problem can be expressed in terms of the trace of $\varvec{G}$:

$$\begin{aligned} \begin{aligned}&\sum _{i,j=1}^n \Vert \varvec{x}_i-\varvec{x}_j \Vert ^2 \,=\, \sum _{i,j=1}^n (g_{ii}-2g_{ij}+g_{jj}) \\&\qquad \qquad =\, 2n\sum _{i=1}^n g_{ii} \,=\, 2n \mathrm {tr}(\varvec{G}), \end{aligned} \end{aligned}$$

(5)

where the second equality follows from (4). Hence, since a matrix is Gramian if and only if it is positive semidefinite, the unfolding problem is equivalent to the following semidefinite programming problem (we refer to [8] for an introduction to semidefinite programming):

$$\begin{aligned} \begin{aligned}&\text {maximize}\; \mathrm {tr}(\varvec{G})\; \text {subject to} \\&\; {\left\{ \begin{array}{ll} \, {\varvec{G}} \,\,\text {is positive semidefinite} \\ \, \forall \{i,j\} \in \mathcal {I}, \; g_{ii} - 2g_{ij} + g_{jj} = \Vert \varvec{v}_i-\varvec{v}_j \Vert ^2 \\ \, {\displaystyle \sum _{i,j=1}^n g_{ij} \,=\, 0.} \end{array}\right. } \end{aligned} \end{aligned}$$

(6)

In our experiments, we use the semidefinite programming solver described in [9]. We now describe how to compute a solution $(\varvec{x}^*_1,\ldots ,\varvec{x}^*_n)$ to the original unfolding problem (2) from a solution $\varvec{G}^*$ to (6). Since $\varvec{G}^*$ is positive semidefinite, its eigenvalues $\lambda _1,\ldots ,\lambda _n$ are nonnegative and

$$\begin{aligned} \varvec{G}^*\,=\, \varvec{P} \, \mathrm {diag}(\lambda _1,\ldots ,\lambda _n) \, \varvec{P}^T, \end{aligned}$$

(7)

where $\varvec{P}$ is the orthogonal matrix whose columns $\varvec{p}_1,\ldots ,\varvec{p}_n$ are the eigenvectors of $\varvec{G}^*$ associated with $\lambda _1,\ldots ,\lambda _n$. Therefore the coefficients of $\varvec{G}^*$ are given by

$$\begin{aligned} g^*_{ij} \,=\, \sum _{k=1}^n \lambda _k p_{ik} p_{jk} , \end{aligned}$$

(8)

where $p_{ik}$ is the ith component of $\varvec{p}_k$. Equivalently, $\varvec{G}^*$ is the Gram matrix of the vectors $\varvec{w}_1,\ldots ,\varvec{w}_n$ defined by

$$\begin{aligned} \varvec{w}_i \,:=\, \textstyle \big ( \sqrt{\lambda _1} p_{i1} ,\ldots , \sqrt{\lambda _n} p_{in} \big ). \end{aligned}$$

(9)

Let $\pi$ be a permutation of $\{1,\ldots ,n\}$ that arranges the eigenvalues in decreasing order, i.e., $\pi$ is a bijection from $\{1,\ldots ,n\}$ to itself such that $\lambda _{\pi (1)} \geqslant \cdots \geqslant \lambda _{\pi (n)}$. Let $\varvec{w}_{i,\pi }$ denote the vector obtained by permuting the components of $\varvec{w}_i$ using $\pi$. If the triangle mesh $(\mathcal {V},\varvec{A})$ lies near a 2-D manifold, then $\lambda _{\pi (2)} \gg \lambda _{\pi (3)}$ [7], and thus

$$\begin{aligned} \varvec{w}_{i,\pi } \,\approx \, \textstyle \big ( \sqrt{\lambda _{\pi (1)}} p_{i\pi (1)}, \sqrt{\lambda _{\pi (2)}} p_{i\pi (2)}, 0, \ldots , 0 \big ). \end{aligned}$$

(10)

Hence each point $\varvec{x}^*_i$ is obtained by keeping the first two components of $\varvec{w}_{i,\pi }$, and so the 2-D embedded mesh is $((\varvec{x}^*_1,\ldots ,\varvec{x}^*_n),\varvec{A})$ with

$$\begin{aligned} \varvec{x}^*_i \,:=\, \textstyle \big ( \sqrt{\lambda _{\pi (1)}} p_{i\pi (1)}, \sqrt{\lambda _{\pi (2)}} p_{i\pi (2)} \big ). \end{aligned}$$

(11)

The quality of this embedding depends on the accuracy of the approximation (10) and can be measured via the ratio $\lambda _{\pi (3)}/\lambda _{\pi (2)}\in [0,1]$; indeed, the smaller this ratio, the closer $(\mathcal {V},\varvec{A})$ to a 2-D manifold. Hence the following quality measure:

$$\begin{aligned} \text {Embedding accuracy}\,\, (\%) \,:=\, 100(1 - \lambda _{\pi (3)}/\lambda _{\pi (2)}). \end{aligned}$$

(12)

3 Results

3.1 Synthetic Examples

We start from a 3-D triangle mesh of the volume swept out by moving the 2-D “C” shape shown in Fig. 2(a) along the z-axis. The mesh is displayed in Fig. 2(b) and was generated using the iso2mesh software (available at http://iso2mesh.sourceforge.net) [10]. We consider the mesh subsets shown in Fig. 2(c) and (d), which can be viewed as meshes of simply- and doubly-curved bands and are thus close to 2-D manifolds.

Fig. 2.

Synthetic data: (a) section of the volume used to generate the synthetic mesh; (b) the whole synthetic mesh; (c) simply-curved band (consisting of the outward part of the mesh); (d) doubly-curved band (obtained by removing the vertices located at the extremities of the “C” shape and the vertices with z-coordinate smaller than 10). (The distance unit is arbitrary.)

Fig. 3.

Embeddings of the simply-curved band and the doubly-curved band shown in Fig. 2(c) and (d). The mesh triangles are colored according to the percentage decrease in area accompanying the unfolding process.

Simply-Curved Band. Figure 3(a) shows the 2-D embedding of the simply-curved band shown in Fig. 2(c). The color scale reflects the percentage decrease of the triangle areas resulting from the unfolding process. Apart from a few artifacts on the border, the triangle areas are well preserved (the median area-decrease is 3.31%). The embedding accuracy (see (12)) is $99.06\%$, which confirms that the 3-D mesh has an intrinsic dimensionality of two and is faithfully represented by its embedding.

Doubly-Curved Band. The doubly-curved band shown in Fig. 2(d) mimics a high-curvature region of the nasal surface at a subunit junction. Its 2-D embedding is displayed in Fig. 3(b). The median area-decrease, 10.91%, is higher than for the simply-curved band. The shrinking of the triangle areas occurs on or near the junction, which appears as a thin strip in the middle of the unfolded band. The two sides separated by this strip correspond to simply-curved bands and hence are well preserved by the unfolding process. The overall embedding accuracy is 98.99%, confirming the two-dimensionality of the 3-D mesh and the faithfulness of its unfolded representation.

3.2 Unfolding a Real Nasal Surface

Data. The real nose mesh (see Fig. 4(a)) is extracted after imaging the face of a healthy volunteer using an optical 3-D scanner (FaceSCAN$^{\text {3D}}$, 3D-Shape, Erlangen, Germany). For simplicity, the inner nostril regions are not considered in unfolding. We also printed a 3-D model of the whole nose (see Fig. 4(b)) for physical template quality evaluation.

Embedded Mesh. Figure 5(a) shows the embedding of the 3-D nose mesh. As in Sect. 3.1, the color scale reflects the percentage decrease of the triangle areas accompanying the unfolding process. The median area-decrease, 10.25% ($0.12\,\text {mm}^2$), is similar to that observed for the doubly-curved band. The shrinking of the triangle areas largely occurs at subunit junctions; the total area-decrease is 14.47% (435.26 mm$^2$). The embedding accuracy is 97.11%, which indicates that the 3-D nose mesh is close to a 2-D manifold and is properly unfolded.

Fig. 4.

Real nasal data: (a) nose mesh obtained from optical 3-D scanning a volunteer’s face; (b) 3-D print of the nose mesh.

Fig. 5.

Templates obtained aftering unfolding the nose mesh shown in Fig. 4(a): (a) digital template; (b) paper print; (c) aluminum foil template delineated from the paper template.

Fig. 6.

Template quality evaluation. The aluminum foil template is wrapped around the 3-D printed model; the arrows point out the regions that are not properly covered.

Physical Template. To create a ready-to-use template, we first scale the 2-D embedding in Fig. 5(a) so that its printing has the same area as the input 3-D mesh in Fig. 4(b). The resulting paper template is then used to delineate the contour of an aluminum foil, as illustrated in Fig. 5(b) and (c). The quality of the foil template is evaluated by wrapping it around the 3-D printed nose model (see Fig. 6). The borders of the 2-D template fit with the 3-D nose. A few small regions pointed out by the arrows are not covered by the template. The nostril holes are not covered, which complies with the input 3-D mesh. The foil is slightly broken in the nasion region (blue arrow), which is predictable from the percentage decrease in area visualised in Fig. 5(a). However, this defect and the non-covered locations on the border are small enough to be compensated by skin elasticity.

4 Conclusion

A nasal cover of correct shape and dimensions is necessary to give a natural appearance to the reconstructed nose. With this in mind, we have developed a mesh unfolding framework to find optimal 2-D skin templates for nasal reconstruction. Starting from a 3-D nose mesh, our approach generates a digital 2-D skin template that eases the preparation of physical 2-D templates (e.g., paper and foil templates). We demonstrated its effectiveness on both synthetic and real nasal surface meshes.