Persistence Based on LBP Scale Space

Abstract

This paper discusses the connection between the texture operator LBP (local binary pattern) and an application of LBPs to persistent homology. A shape representation - the LBP scale space - is defined as a filtration based on the variation of an LBP parameter. A relation between the LBP scale space and a variation of thresholds used in the segmentation of a graylevel image is discussed. Using the LBP scale space a characterization of (parts of) shapes is demonstrated based on simple shape primitives, the observations may also be generalized for smooth curves. The LBP scale space is augmented by associating it with polar coordinates (with the origin located at the LBP center). In this way a procedure of shape reconstruction based on the LBP scale space is defined and its reconstruction accuracy is demonstrated in an experiment. Furthermore, this augmented LBP scale space representation is invariant to translation and rotation of the shape.

1 Introduction

Biasotti et al. note in [1] that a digital model of an object is quantitatively similar to the object, while a description is only qualitatively similar. The authors further quote a clear distinction between representation and description:

“An object representation contains enough information to reconstruct (an approximation to) the object, while a description only contains enough information to identify an object as a member of some class.” [1, p. 5]

Following this definition we study in this paper a novel shape representation based on topological persistence and on local binary patterns (LBPs): the LBP scale space. We show that an approximate reconstruction of the shape is possible using the LBP scale space. The shape representation is given as a vector of the persistence of LBP types around a center pixel. It may as such also be used to classify and compare shapes and can thus also be seen as a topological shape descriptor.

Shape description based on topological persistence is for example defined by the size functions described by Verri et al. [2] which represent the persistent Betti number $\beta _0$. Carlsson et al. presented persistence barcodes for shape description and classification [3]. For shape retrieval the shapes may for example be compared based on their persistence diagram using the matching distance as presented by Cerri et al. [4]. Barcodes encode the persistent homology of a data set in the form of a parametrized version of a Betti number [5]. Barcodes visualize the lifetime for which a features persist and therefore encode multisets of intervals in $\mathbb {R}$. While a persistence diagram is “a multiset of points (u, v) whose abscissa and ordinate are, respectively, the level at which a new k-homology class is created and the level at which it is annihilated through the filtration” [4, p. 2]. This filtration produces a sequence of nested spaces. While the filtration grows, topological features appear (birth) or disappear again (death). The interval between the birth and the death of such a feature, its lifetime, is its persistence. A filtration is for example given by the level cuts of a Morse function on a manifold, as it is used to derive a Reeb graph (another topological shape representation) [6]. The simplest way to obtain a Reeb graph of an image or a 3D shape is to use a height function as a Morse function. In the same way persistence diagrams or barcodes can be determined using a filtration based on a height function. Such topological shape representations based on a filtration are in general dependent on the filtration. Especially height functions are not invariant to rotations of the shape and therefore lack in representational power.

The proposed LBP scale space represents a shape based on changes of the local topology captured by LBPs of increasing radii¹. Since the LBP takes a circle around a center pixel into consideration, this representation is invariant to translation and rotation of the shape. The LBP scale space can be further extended by associating polar coordinates with the observed local topology. In this way the LBP scale space provides the possibility of reconstruction of the shape (for a discrete LBP scale space up to the sampling of the scale space) based solely on this shape descriptor.

The rest of the paper is structured as follows: Sect. 2 describes the way in which the LBP texture operator captures local topology. In Sect. 3 filtrations using LBPs are presented. The LBP operator is used to derive a shape descriptor based on its persistence - the LBP scale space. Two experiments were conducted: a study of the behaviour of the LBP scale space for special shapes (primitives) and an approach to shape reconstruction based on the LBP scale space. These experiments and the results are discussed in Sect. 4. Section 5 concludes the paper and gives an outlook to future work.

2 Capturing Topology Using LBPs

Although originally proposed as a tool of texture classification, LBPs have in the past also been studied as tool of topological shape and image analysis. For a given grayscale digital image I, the local-binary-pattern codification of I: LBP(I) again yields a grayscale digital image. The grayvalues of LBP(I) are now LBP codes that are used to represent the texture element at each pixel in I. However, the LBP codes not only capture local texture information, they also describe the local topology observed.

2.1 Introduction to LBPs

LBPs were first introduced for texture classification [7] and since then became popular texture operators. An LBP computation around a pixel $p=(x,y)$ studies the grayvalues along a subsampled circle of a specific radius r around p. Each position in a bit pattern (corresponding to the sampling points along the circle) is set to 1 if the grayvalue at the sampling point ($g(x_i)$) is larger than or equal to the grayvalue of the center pixel (g(p)) and to 0 otherwise (Fig. 1a and b):

$$\begin{aligned} s(x_i)= {\left\{ \begin{array}{ll} 1 &{} \text {if } |g(p)-g(x_i)| \ge 0\\ 0, &{} \text {otherwise} \end{array}\right. } \end{aligned}$$

(1)

The two parameters P and r determine the LBP computation: P fixes the number of sampling points along a circle of radius r around the center pixel, for which the LBP operator is computed [8]. Figure 1c shows different parameter configurations. By varying r we may examine concentric circles around p, the LBP is given by c(x, y, r).

Fig. 1.

(a) and (b) LBP computation for center pixel p and (c) variations of the parameters P (sampling points) and r (radius).

2.2 Connected Components of a Graylevel Image

A binary segmentation of a graylevel image is easily obtained as level sets using a threshold t or an interval around the threshold t:

$$\begin{aligned} |g(x,y)-t|\le \epsilon \end{aligned}$$

(2)

Such a threshold interval is also used in the LBP computation of robust local binary patterns (RLBPs) [9]:

$$\begin{aligned} s(x_i)= {\left\{ \begin{array}{ll} 1 &{} \text {if } |g(p)-g(x_i)+t| \ge 0\\ 0, &{} \text {otherwise} \end{array}\right. } \end{aligned}$$

(3)

According to Eq. (2) the image is then segmented into several connected components $C_i, i=1,...,n$. One such connected component $C_a$ consists of either:

• 1. grayvalues that are inside the defined interval around $t\pm \epsilon$ so called plateaus:

  $g(x,y)\in C_a: t-\epsilon \le g(x,y) \le t+\epsilon$,

• 2. grayvalues that are larger than the threshold interval $t + \epsilon$ so called maxima:

  $g(x,y)\in C_a: g(x,y) > t+\epsilon$,

• 3. or grayvalues that are smaller than the threshold interval $t - \epsilon$ so called minima:

  $g(x,y)\in C_a: g(x,y) < t-\epsilon$.

The input image is thus segmented into connected components, each of them belonging to one of the mentioned categories: plateau, minimum and maximum. The connected components (which may have holes) are surrounded by closed boundaries b(x, y).

2.3 Local Topology Based on LBP Types

In previous work we defined a shape descriptor based on persistence of LBP types around critical points of a shape [10]. For this purpose we analysed a binary shape based on LBPs. We computed LBP types which describe the local topology of the foreground region around the center pixel p. The LBP types are defined by the number of transitions from 0 to 1, respectively vice versa (bit-switches) in the LBP code.

• (local) maximum (no bit-switches: the bit pattern contains only 0s),

• (local) minimum (no bit-switches: the bit pattern contains only 1s),

• plateau (no bit-switches: the bit pattern contains only 1s, but all pixels of the region have the same gray value),

• slope (two bit-switches - compare uniform patterns [8]),

• saddle point (four or more bit-switches) [11].

In a segmented image these bit-switches BS correspond to the intersections of the boundary of the connected component which holds p and the LBP circle of radius r:

$$\begin{aligned} BS = b(x,y) \cap c(x,y,r) \end{aligned}$$

(4)

Note that the LBP types not necessarily correspond to the types of connected components as defined in the previous section.

3 LBP Based Persistence

In persistence, those features which persist for a parametrised family of spaces over a range of parameters are considered signals of interest. Short-lived features are treated as noise [5]. The persistence of a feature is given as its lifetime, the span in between the birth and the death of a feature according to a filtration of a space which is “a filtered space, a nested sequence of subspaces that begins with the empty and ends with the complete space” [12, p. 5].

Fig. 2.

Increasing the radius r for a fixed center p shifts the intersections with b along the boundary b (blue arrows). Varying $\epsilon$ moves the intersections along the LBP circle (orange arrows). (Color figure online)

3.1 Filtration Based on LBPs

Using LBPs we may perform a filtration either:

• 1. by varying the radius r of the LBP computation c for a fixed boundary b,

• 2. or by varying the parameter $\epsilon$ of the segmentation thus varying b, for a fixed r.

Fig. 3.

Configurations of intersections observed when varying the radius allow assumptions about a shape’s connected components.

Fig. 4.

Characterisation of shape parts based on the LBP scale space.

Varying the radius r of the LBP computation c corresponds to a movement along the boundary b (blue arrows in Fig. 2), while varying $\epsilon$ (varying the boundary b) leads to a movement along the circle defined by r and p (orange arrows in Fig. 2). By continuously increasing the radius r the intersection points are moving towards or away from each other along the boundary b. For a certain radius the circle will not intersect the boundary anymore but touch it in one point. By further increasing the radius a pair of intersections will disappear or, ultimately the whole shape will be inside the circle of radius r, this corresponds to an LBP of type maximum.

The persistence of an LBP type can be measured by the movement of the intersection points along the boundary b: For a certain radius r the circle of the LBP computation c intersects the closed boundary b 2n times (apart from osculation points). These intersections divide the shape in $n+1$ regions and 2n boundary segments (in a persistence diagram: a “birth” for each region / segment). By increasing r the intersection points are moving along b. Once two intersection points coincide the LBP type changes (in the persistence diagram the “death” of the respective segment / region).

Assumptions about a connected component’s structure can be made by varying the radius r of the LBP computation c with fixed center p. By analysing the configurations of intersections (bit-switches BS) observed when increasing r a conclusion about characteristics of the shape’s boundary can be drawn (see Fig. 3). Moreover, further connected components in the vicinity can be detected. A connected component is divided into several parts that are either covered by the LBP circle of a certain radius or not. These parts can be characterized as follows (see also Fig. 4):

• 1. inside a shape (interior): The shape is simply connected and corresponds to the bounded connected component of the Jordan curve theorem. Parts inside a shape are identified by intersection points that are moving closer to each other along the boundary for increasing LBP radii until they converge in osculation points.

• 2. outside a shape (exterior): This connected component is unbounded, indicated by intersection points that diverge along the boundary for increasing LBP radii.

• 3. holes: A hole in the foreground connected component can show no intersections with the LBP circle because it is fully contained inside the LBP circle or not at all covered by the LBP circle. A hole that is intersected by the LBP circle needs special consideration: the shape is divided in parts inside and outside the LBP circle. Since the hole is intersected by the LBP circle, the hole is reduced to concavities along the boundaries of these individual shape parts - the topology changes.

Fig. 5.

LBP scale space in the (a) a discrete case showing an example sampling scheme and (b) in the continuous case - the osculation points marked along the red LBP circles are critical points at which the topology changes. (Color figure online)

3.2 LBP Scale Space

The LBP scale space is a novel shape representation proposed in this paper. For a chosen pixel or point of a shape or connected component (as LBP center) we compute the LBP over a range of scales (range of LBP radii). We may start with a radius of 0 and increase it either continuously or in the discrete case according to a predefined sampling scheme (for example increasing always by 1 to cover all integer radii, which corresponds to the full pixel resolution). Figure 5 illustrates the LBP scale space for regular sampling and sampling at critical points.

We are interested in the local topology captured by the LBPs for varying radii. We therefore consider the number of bit-switches observed for each of the LBP radii analysed. This shape representation can be stored as a vector or matrix (depending on the sampling and whether the radii need to be explicitly stored as well). The LBP scale space may be used as a shape descriptor for classification or recognition purposes, as we showed for a similar shape descriptor in [10]. Moreover, the LBP scale space enables the reconstruction of a shape based on this representation. For this purpose, we extend the LBP scale space by polar coordinates. This procedure is discussed in more detail in the experiments Sect. 4.2.

4 Experiments

The conducted experiments include a thorough study of characteristics of the proposed LBP scale space on simple shapes (primitives) and an application to shape representation and reconstruction based on the LBP scale space. As test dataset for the shape reconstruction we use a dataset of binary shapes.

4.1 Shape Analysis - Special Cases

We study two simple shapes (primitives): we start with the simplest shape - a circle, before moving on to an already more generalised shape - an ellipse. For this experiment we study the shape primitives using our LBP scale space with the center pixel of the LBP computation located inside the shape. Based on the LBP scale space derived for varying LBP center locations inside the shape, we categorize parts of the shape according to the LBP scale space. This experiment is related to a shape descriptor based on the LBP scale space that we presented in previous work [10], since it may identify locations within the shape which are well suited as well as parts that may not be suitable at all as centers for this shape description.

This study although conducted on circles and ellipses is not limited to these shapes: By computing the medial axis of a shape and splitting the shape into parts at skeleton branching points we can derive a normalised shape representation by straightening the skeleton segments. Any shape may therefore be represented by circles (according to the medial axis) or more general by ellipses. In addition, we show that some observations made for the shape primitives can be generalised to smooth curves.

Circle: For a circle with radius $r_c$ we can distinguish the following two cases:

• 1. LBP center coincides with center of the shape (circle):

  In this case there are either no intersection points between the LBP circle and the shape (the LBP radius is smaller or larger than $r_c$) or the whole boundary of the shape intersects with the LBP circle (the LBP radius equals $r_c$).

• 2. LBP center at any location within the shape, other than the circle’s center:

  We observe one intersection point $t_1$ for the LBP circle with radius equal to the smallest distance to the boundary and this boundary itself. When increasing the LBP radius we observe two intersection points which move away from $t_1$ until they converge at the second osculation point of the boundary and the LBP circle $t_2$. We thus observe the following sequence of number of intersections: 0-1-2-1-0.

Ellipse: For an ellipse there are four cases to distinguish. We start with the most general case (1) since all others (2–4) are special cases of it:

• 1. The LBP center is located inside the ellipse but not on the ellipse’s axes:

  We observe one intersection point between the LBP circle and the shape’s boundary (LBP radius equal to the smallest distance of the LBP center to the shape boundary). For increasing radii we observe two intersection points which move away from each other and for further increasing radii three intersections (one of them a degenerate intersection). Increasing the radius even further yields four intersections. The intersection points in each case in pairs move away from each other and towards each other along the shape’s boundary, until two of them converge. Here we observe three intersections, for increasing radii again two intersections until these two as well converge in one osculation point for the maximum radius. The sequence of number of intersections therefore is: 0-1-2-3-4-3-2-1-0.

• 2. The LBP center is located at the major axis of the ellipse:

  The smallest distance to the shape’s boundary is smaller than or equal to the radius of the circle of curvature for a major apex: In this case we observe one intersection point between LBP circle and shape’s boundary (at one major apex). For increasing radii we observe two intersection points which move away from one major apex towards the other. The intersection points converge again in the second major apex. Thus, we observe the following sequence of number of intersections: 0-1-2-1-0.

  The smallest distance to the shape’s boundary is larger than the radius of the circle of curvature for a major apex: We observe two osculation points of the shape’s boundary with the LBP circle. For increasing radii we observe four intersection points, which converge pairwisely towards the major apexes. Once two of them converge in an apex, we observe three intersections. For further increasing radii we observe two intersections until these two converge in the second major apex. The sequence of number of intersections is as follows: 0-2-4-3-2-1-0.

• 3. the LBP center is located at the minor axis of the ellipse:

  We observe one osculation point at a minor apex in which the LBP circle intersects the shape’s boundary. For increasing radii we observe two intersections. When increasing the radius further a third intersection at the second minor apex can be observed, followed by four intersections. The intersection points pairwisely move away from the minor apexes until they converge pairwisely in the major apexes and only two intersections remain - the maximum LBP radius is reached. The sequence of number of intersections is: 0-1-2-3-4-2-0.

• 4. the LBP center coincides with the center of the ellipse:

  We observe two osculation points of the shape’s boundary with the LBP circle located at the minor apexes of the ellipse. For increasing radius four intersection points are observed which move away from the minor axis towards the major axis along the shape’s boundary. The intersection points converge in the major apexes. This yields the following sequence of number of intersections: 0-2-4-2-0.

Smooth Curves: Some observations made for circles and ellipses in this experiment apply also for smooth curves in general. For a continuously increasing radius we observe intersection points at the shape’s boundary that move along the boundary and cover the boundary completely when considering the complete LBP scale space. Osculation points are degenerate intersection points since two intersection points coincide at such an osculation point. We can determine the osculation points as the points along the shape’s boundary for which the LBP circle’s and the boundary’s tangent in that point coincide. These osculation points are critical points and describe a birth or death of a component. LBP circles of radii in the interval in between those radii associated with osculation points yield true (non-degenerate) intersections. The interval of radii spanned by such critical points thus describes the lifetime (persistence) of a component.

Fig. 6.

Reconstructions - green, input - red, matching boundaries - yellow. LBP center - white: (a) max. distance transform, (b) min. eccentricity, (c)–(f) chosen manually. (Color figure online)

4.2 Shape Reconstruction

We compute an LBP scale space for a fixed center point inside a shape. The LBP scale space consists of the intersections (bit-switches see Eq. 4) of the shape boundary with the LBP circle for radii ranging from 0 to the radius for which no intersection appears anymore (largest Euclidean distance from the LBP center to the shape boundary). We further associate angles measured around the LBP center with the intersections. In this way we obtain polar coordinates of all intersections and can restore the boundary and thus the shape based on this LBP scale space. For a discrete LBP scale space the quality of the reconstruction is of course dependent on the sampling of the LBP radii and the angle measurement as well as on the chosen center point.

Fig. 7.

Precision of the reconstructed boundary (boundary 1 and 2) and the reconstructed shape (shape 1 and 2) for the 99 shapes in the dataset and two different LBP centers.

This LBP scale space representation is invariant to translation and rotation of the shape. The rotation leads only to a change in the angles associated with the intersection points. All angles measured for a shape are altered by a constant $\alpha$ which complies with the angle of the rotation applied to the whole shape (for the discrete case deviations from $\alpha$ may be observed because of the sampling).

We tested the accuracy of the LBP scale space reconstructions on the Kimia99 dataset [13]. The reconstruction obtained using a discrete LBP scale space is highly dependent on the chosen LBP center. This is well visible in the comparison of reconstructions based on different locations of center points (see Fig. 6).

For the experiments on the whole dataset, we used two different LBP centers. We defined the LBP center for each shape as (1) the location of minimum eccentricity [14] and (2) the location of maximum distance transform. We used a regular sampling scheme of 1 pixel radius increases, thus covering all integer radii in the range of the LBP scale space of a shape. Based on the LBP scale space obtained for a shape, we reconstructed the shape again. An input shape together with the reconstruction based on the LBP scale space is shown in Fig. 6. For all shapes in the dataset we evaluated the reconstruction quality for the boundary only and for the whole shape. For this purpose we computed the reconstruction error considering the set of pixels: $M_b$ which are the boundary pixels of the reconstruction matching the boundary pixels of the input shape respectively $M_{shape}$ all shape pixels of the reconstruction matching the input shape - the true positives. We computed the precision using the set of boundary pixels of the input shape $b_{in}$ and as well as the set of pixels of the whole input shape $shape_{in}$:

• 1. precision boundary: $M_b / b_{in}$

• 2. precision shape: $M_{shape} / shape_{in}$

Figure 7 shows the precision of the reconstruction of the boundary and the whole shape for all 99 shapes in the dataset. The precision for the boundary is low, due to the fact that deviations from the exact position of boundary pixels even by only 1 pixel in the reconstruction are considered an inaccurate reconstruction. The boundary precision is at maximum 0.67 for the center at the location of maximum distance transform (Fig. 7: boundary1) and maximum 0.72 for the center at the location of minimum eccentricity (Fig. 7: boundary2). Since the goal of this approach is to reconstruct the whole shape not only its boundary, we used morphological operations, to close gaps in the boundary and to fill the shape region. The precision for the whole shape is considerably higher: up to 0.99 for the LBP center at the location of maximum distance transform (Fig. 7: shape1) and 1.00 for the LBP center at the location of minimum eccentricity (Fig. 7: shape2). As visible in Fig. 7 the precision of the reconstructed shape is low for some shapes of the dataset - in these cases the boundary reconstruction showed larger gaps, which did not allow to reconstruct a connected boundary that could be filled. In general the reconstruction of the whole shape works well: a precision of 0.9 is reached for 92 % of the shapes for the LBP center at the maximum distance transform and for 81 % of the shapes for the LBP center at the location of the minimum eccentricity.

5 Conclusion and Future Work

The novel shape representation - the LBP scale space - is based on persistence, studying a shape by creating a filtration using the LBP operator over a range of scales (radii). The experiments presented in this paper show that the LBP scale space may be easily augmented using polar coordinates around the LBP center. Based on this extended shape representation not only classification of a shape but also a reconstruction is possible.

In future work we would like to perform experiments including noise flawed input shapes, since we can employ the persistence information to exclude noise from the reconstruction. The LBP scale space may also be extended to the 3D space using spheres instead of circles. Of course the intersection points used for the LBP scale space in 2D are also extended to intersection curves between the LBP sphere and the 3D shape then. Moreover the presented approach may be used in future work as a tool of quality control in obtaining binary image segmentations, similar to the MSER (maximally stable extremal regions) approach [15]. The segmentation is defined by the threshold t as well as the interval defined by $\epsilon$. By fixing the LBP parameter r and varying the boundary b we may for example determine parameters t and $\epsilon$ that yield a segmentation of a given number of connected components with stable boundaries regarding the persistence of LBP types when varying the radius r of the LBP computation c slightly.