Scale-Insensitive MSER Features: A Promising Tool for Meaningful Segmentation of Images

Abstract

Automatic annotation of image contents can be performed more efficiently if it is supported by reliable segmentation algorithms which can extract, as accurately as possible, areas with a certain level of semantic uniformity on top of the default pictorial uniformity of regions extracted by the segmentation methods. Obviously, the results should be insensitive to noise, textures, and other effects typically distorting such uniformities. This chapter discusses a segmentation technique based on SIMSER (scale-insensitive maximally stable extremal regions) features, which are a generalization of popular MSER features. Promising conformity (at least in selected applications) of such segmentation results with semantic image interpretation is shown. Additionally, the approach has a relatively low computational complexity $(O(log n\times n)$ or $O(log n\times n\times log(log(n)))$, where n is the image resolution) which makes it prospectively instrumental in real-time applications and/or in low-cost mobile devices. First, the chapter presents fundamentals of SIMSER detector (and the original MSER detector) in gray-level images. Then, relations between semantics-based image annotation and SIMSER features are investigated and illustrated by extensive experiments (including color images, which are the main area of interest).

Appendix

The Appendix contains details of computational steps in SIMSER detection, focusing on the prospective hardware or hardware-supported implementations. However, such details cannot be fully explained without an insight into the detection of MSER features. Thus, the included information (a summary of the results presented in [22]) covers most important facts on architectures used in MSER and SIMSER detection, as well as architectures specifically proposed for SIMSER detection only.

Detection of local minima in the threshold space

At each threshold level, the binary image of $M \times N$ size is represented by three data structures:

• Seed matrix of regions SM (of the same size as the image) with the initial content $SM_{i,j}= M\times (i-1)+j$, i.e. each pixel is a seed for itself. After processing, $SM_{i,j}=K$, where K indicates the initial pixel (seed) of the region to which (i, j) pixel belongs.

• Region Size matrix RS (of the same size) specifying the size of region to which each (i, j) pixel belongs. Initially, $RS_{i,j}=1$, i.e. each pixel is a separate region of unit size.

• Map-of-regions array, which for each image region lists its seed, the binary color and the size.

A small binary image and the final contents of its SM and RS matrices are shown in Fig. 3.8, while its Map-of-regions is given in Table 3.2.

Fig. 3.8

A small binary region and its final SM and RS matrices

Table 3.2 Map-of-regions for Fig. 3.8 image

Given such representations for the sequence of binary regions over three neighboring threshold levels (note that such regions are always nested) the local minima of $q_Q$ (see Eq. 3.2) and $qt_Q$ (see Eq. 3.4) growth-rate functions can be straightforwardly identified. In other words, MSER regions can be detected or SIMSER candidates (i.e. the regions which satisfy the local minimum criterion in the threshold space) can be pre-selected.

Detection of local minima in the scale space

To identify SIMSER blobs, the regions pre-selected as the local minima in the threshold space should also be confirmed as the local minima in the scale space, i.e. the minima the second growth-rate function $qs_Q$ (see Eq. 3.5). To verify this, two operations are needed:

• The original input image should be repetitively processed by a smoothing filter. This is just a convolution with the filter kernel, i.e. the operation which can be straightforwardly into hardware. Its computational complexity is O(n).

• The correspondences between binary regions in the neighboring scales should be established and, based on that, the values of $qs_Q$ growth-rate evaluated. This is not a straightforward operation because binary regions over a sequence of scales often do not nest (a simple example is shown in Fig. 3.9).

Fig. 3.9

An example of not nested (overlapping) black and white regions over two neighboring scales (smoothing removes sharp extremes, both dark and bright)

To solve this problem, the following pseudocode is proposed (its less effective variant which, nevertheless, clearly indicates O(n) complexity of the algorithm was given in [21]):

Evaluation of $qs_Q$ growth-rate function

The scheme takes two binary images (at the same threshold but at the neighboring scales) their RS and SR matrices, and their maps-of-regions (see above). For each binary region at the current scale, the identifier of the next-scale region is found, and the value of the growth-rate function $qs_Q$ is evaluated. Therefore, the changes of $qs_Q$ can be tracked over the scales, and the local minima can be easily found.

In this way, all operations needed to identify SIMSER features are completed.

As an example, a pair of binary images from two neighboring scales is shown in Fig. 3.10, and the corresponding results of the above operations are included in Table 3.3. In this example, Region 4 has the best chance to be a local minimum (with the smallest value of $qs_Q$). To confirm that, however, similarly computed values of $qs_Q$ for Region C (which is the correspondence of Region 4 in the next scale) and for the corresponding region in the previous scale, should be larger (Fig. 3.11).

Fig. 3.10

Computing $qs_Q$ growth-rate function in the scale space. The left image is in the current scale, while the right one in the next scale

Table 3.3 $qs_Q$ processing for Fig. 3.10 images. MX values are the region intersection sizes

Altogether, it can be concluded that SIMSER detection architecture is a relatively simple extpansion of the MSER detection architecture, so that hardware implementation of SIMSER detector is a feasible task.

Fig. 3.11

Original images (a,b,c) and the SIMSER-based segmentation results obtained from: grey-level copies (d,e,f), red (g,h,i), green (j,k,l) and blue (m,n,o) channels