1 Introduction

The detection of bubbles, or more generally, transparent roughly spherical objects in a liquid is important for several machine vision applications, for example, dispersion of oil drops in water [10], air bubble detection in dense dispersion [34], and air bubble segmentation from the high-speed video acquired by equipment of multi-phase flow device [30]. In this paper, the research is driven by the pulp making industry. The appropriate moment for terminating the pulp bleaching can be determined by estimating the gas volume, and the size and number of detected bubbles in the pulp suspension [9].

The bubble detection problem is not easy to solve because the bubbles are transparent and the illumination conditions are challenging when imaging in an industrial process. These factors cause the bubble appearance to vary from a pair of ring-like, bright ridge edges to blurred dark edges with contrast reversal and multiple interreflections. Oriented filter responses caused by the bubbles form a set of concentric arcs with a common center point as shown in Fig. 1. In this work, the set of concentric circular arcs is noted as a concentric circular arrangement (CCA) that is modeled by an annulus parametrized by its radius, center, and annulus width. The bubble detection problem is formulated as a search for CCAs and is solved in a hypothesize–optimize–verify framework.

Fig. 1
figure 1

Examples of images of bubbles (top row) and binarized local maxima of oriented ridge filters (bottom row)

Full size image

A preliminary version of this research appeared in the conference paper [27], where we showed through initial experiments that the proposed method accurately estimates the gas volume. This work extends the work by including (i) a review of related works, (ii) experiments on an extended set of pulp suspension images with the ground truth, and (iii) experiments on a set of oil dispersion images to study the applicability of the method to solve similar problems. The main contributions of the paper are as follows: (i) a thoroughly evaluated method for the detection of bubble-like objects in liquid as CCAs and (ii) the application of the method to bubble detection in pulp suspension. Together they enable the machine-vision-based gas volume estimation based on imaged pulp suspensions.

The paper is organized as follows: Sect. 2 discusses the existing solutions for the similar tasks. The method for CCA detection is presented in Sect. 3. Experimental evaluation of the method in terms of bubble detection accuracy and gas volume estimation is described in Sect. 4. The conclusions are given in Sect. 5.

2 Related work

Bubbles or drops manifest themselves as roughly circular objects, which motivates the researchers to solve the problem as the detection of circles. There are two common approaches that are used to detect circular objects: geometry-based and appearance-based approaches. In the geometry-based approach, a circular model parameterized by its center and radius is fitted to the image edge map. In the appearance-based approach, a template of a bubble is created, the image is convolved with the template, and the local maxima of the convolution are sought.

2.1 Geometry-based approaches

The geometry-based approaches typically utilize a voting technique, such as the Hough Transform (HT) [11] or its modifications [16]. The HT for circle detection [33] describes a circle by its center coordinates \((x_c, y_c)\) and radius r. Each point (x, y) in an image corresponds to a circular cone in the \((x_c, y_c, r)\) parameter space. If points lie on one circle, their cones in the parameter space intersect in one point that describes the center coordinates and the radius of the circle.

To address the main weaknesses of the HT, such as computational complexity and storage requirements, several revisions of the basic HT have been proposed (see e.g., [15]). The Randomized and Probabilistic HTs utilize the random sampling techniques to sample the hypothesis of the circles from the edge pixels and remove verified hypotheses from the image to accelerate the process. HT and its modifications have been utilized in several applications, e.g., segmentation of spherical particles in transmitted light image stacks [23], segmentation of bubbles and drops in complex dispersions in bioreactors [28], and detection of circular objects in pulsative medical video [22].

Alternatively to HT, also stochastic approaches and evolutionary based techniques have been proposed in order to reduce computational time and memory requirements. For example, in [2], a genetic algorithm to select circle candidates was implemented. The method, however, does not perform well with circles with irregular shape. In [8], an adaptive bacterial foraging optimization for circle detection was introduced. The method produces good detection results but is sensitive to noise.

A method proposed in [20] utilizes the gradient information to produce a projection line in the \((x_c, y_c, r)\) space from each edge pixel. If the projection lines of two pixels intersect in one point, they belong to the same circle. If the projection lines of two pixels do not intersect, they produce a hypothesis of a circle. The likelihood of each such hypothesis is modeled as a Gaussian distribution dependent on the distance between the two projection lines, equal to the length of a line perpendicular to the both projection lines. The score of a hypothesis is computed by marginalization over all edge point pairs that contribute to the hypothesis. The method succeeded in human iris detection.

Taboada et al. [28] presented a semi-automatic image analysis method for the segmentation of bubbles and drops. In the method, false positives were manually removed after the HT. The work was continued in [10] where the focus was on the detection of oil drops in dispersion images and automatic elimination of the false positives. Since the oil drops were not the only circular objects in the images, the authors designed a feature to distinguish between the oil drops and other objects. They exploited the average value of the difference between the exterior and the interior gradient orientations with respect to the drop border. The classification procedure utilizing the designed feature was applied after the HT, and the classifier was trained by using the preselected oil drop hypotheses generated by the HT.

In the method presented in [7], a random line generated in an input image intersects with two pixels producing a hypothesis of a circle. The HT accumulator space is incremented with the value which measures the degree of matching between the gradient directions in the sampled points. The method demonstrated good results on synthetic and two real world images.

2.2 Appearance-based approaches

The appearance-based approach works with grayscale images, or in some cases, with edge-maps. The approach is based on the sliding window approach [29] where a template of the object of interest is created and the grayscale image is convolved with the template. In order to find the objects, the maxima of the filtration responses are searched. An appearance-based approach for the bubble detection has been employed in [34]. They improved the standard appearance-based approach by convolving the image at multiple scales, by utilizing multiple prototypes (templates) of a bubble, and by detecting the rotated prototypes. Based on the visual evaluation of the results, the method outperformed the standard HT. However, the authors reported a relatively large computational time for the correlation step.

In [18], the performance of an appearance-based drop detection algorithm was improved by preprocessing the image using illumination correction techniques. After template matching, the found drop candidates were further verified. In [31], a raindrop detection algorithm was proposed. The method starts with a raindrop candidate detection using CHT-based shape features combined with color and texture based saliency maps and proceeds to learning-based raindrop verification.

2.3 Other approaches

The problem of cell segmentation shares common characteristics with bubble detection. Methods developed for cell segmentation include intensity thresholding, morphological operations, region growing, feature detection and deformable model fitting [19]. There are, however, some important differences between bubble detection and cell segmentation problems: the cells have more complex shapes while typical bubbles are close to perfect circles, cells have smaller variation in size assuming only certain cell type is considered, a small bubble can appear fully inside larger bubble, and bubbles are more transparent.

The cell detection is often solved as blob detection problem. In [4], a method to detect blobs defined as small round regions in an image is proposed. The detection is carried out by solving a region segmentation problem where the blobs form one region and the areas between blobs form the other. In [1], a similar blob detection approach is further developed by proposing a more complex statistical model that uses detected blobs as object candidates. The blob detection approach, however, cannot be applied to bubble detection as such since bubbles typically have distinguishable edges but are otherwise transparent. Therefore, bubble detection should be based on edge points instead of detection of uniform blobs.

The outer boundary segmentation of embryos is another similar problem. An embryo has a circular shape similar to bubble. For example, direct least square fitting of ellipses have been proposed to solve the problem [5, 12]. However, due to the fact that the images typically contain only embryo, these methods are not generalizable to images containing multiple bubbles.

Fig. 2
figure 2

Examples of the pulp suspension images

Full size image

In [35], the segmentation of overlapping bubbles was approached using concave points extraction through the polygonal approximation and ellipse fitting. The method was, however, developed for practically noise-free images where bubbles were clearly distinguishable from the background. A similar approach was proposed in [3] where the problem of incomplete cells with elliptical shapes was addressed by combining certain heuristics with concave point extraction and ellipse fitting.

Also deformable models have been used in circular object segmentation to address the irregularities in the shape (see e.g., [14]). These methods are, however, very sensitive to initialization.

2.4 Drawbacks of the existing methods

The existing solutions for the circular object detection do not provide a unified framework for the bubble-like object detection. The Circular Hough Transform (CHT) approaches suffer from a large number of false positives and are sensitive to noise. Moreover, they do not take into account the appearance of the bubbles as an object with multiple light interreflections. Template matching techniques are difficult to apply for the bubble detection since depending on the bubble location and lighting, the bubbles in the suspension appear differently (bright ring-like edge or dark moon-like shape as shown in Fig. 2). In addition, if the depth of focus in imaging is large, the bubbles may overlap in the 2D image. Moreover, increasing the number of templates dramatically increases the computation time. The concave point detection based approaches are developed for images with a low amount of noise and cannot be generalized to industrial bubble images. In this work, we propose a novel method for bubble detection to address these shortcomings.

Fig. 3
figure 3

CCA: a The CCA model; b The weight function \(q(r_a)\) for the edge distance from the center; c Average CCA support at the different distances from the center

Full size image

3 The CCA detection method

3.1 CCA

CCA \(H = (\mathcal {A}, c, R, \theta )\) is a set of concentric circular arcs \(\mathcal {A}\) with a center \(c = (x_c, y_c)\) located within an annulus of radius R and width \(2\theta \) (see Fig. 3a). An arc is a connected component of local maxima of oriented filtered responses \(A = (P, r_a)\) parameterized by a pixel list P and a radius \(r_a\). The score of a CCA is comprised by the contribution of separate pixels located in the CCA area. The contribution of a pixel \(p \in A\) with an orientation \(\alpha _p\) (degrees) to the support of a CCA hypothesis with a center c is computed as

$$\begin{aligned} f_{p} = g(\Delta \alpha ) \cdot q(r_a) \end{aligned}$$
(1)

where \(\Delta \alpha = |\alpha _p - \arctan (|x_{c}-x_{p}|/|y_{c}-y_{p}|)| \) describes if the orientation of the pixel is consistent with the CCA model (see Fig. 3a). The contribution \(f_p\) models the probability of p belonging to the CCA assuming the pixel independence. The weight function for the pixel orientation \(g(\Delta \alpha ) = \frac{1}{1+\Delta \alpha }\) puts less weight to the edges with inconsistent orientation. The function \(q(r_a)\) weights the pixel input depending on its location within the CCA area. The weight function \(q(r_a)\) for the edge distance from a center is presented in Fig. 3b. The weight function \(q(r_a)\) was designed in such way that no weight is put to the pixels outside the CCA area, the pixels within the ring of width wR are put weight one and the rest of inner pixels get quadratically decreasing weights normalized between zero and one. Thus, the method is trying to maximize the radius of the CCA to include the whole bubble structure. The parameter w, width of the average bubble edge was obtained by computing an average CCA support for manually annotated bubbles at different distances from the CCA center (see Fig. 3c).

The CCA detection is illustrated in Fig. 4 outlined in Algorithm 1. After the oriented edge map is computed, the CCA hypotheses are generated by sampling from the edge map, optimized with respect to radius and center, and selected based on their score.

Fig. 4
figure 4

The proposed method for bubble detection

Full size image
figure f

3.2 Oriented edge map computation

Since bubbles manifest themselves as objects with ridge edges, we adopted the solution for ridge detection from [17]. The images are filtered by a set \(\mathcal {M} = \{\mathbf {M_j}\}\) of second derivative zero-mean oriented Gaussian filters in eight directions. For further processing the absolute value of the responses is taken to capture both bright and dark edges. The dominant orientation \(\alpha \) of the edge normal in each point is computed as the maximum of the eight filter responses.

After the orientation of each edge pixel has been computed, the non-maximum suppression [6] is performed. If a filter response of a pixel is a local maximum in the dominant orientation, its value is set to one and the neighboring pixels in the dominant orientation to zero. The values with the filter response lower than the threshold \(T_{filt}\) are also set to zero.

3.3 CCA hypothesis generation, optimization, and selection

After the edge map \(\mathbf {E}\) is computed a set of arcs \(\mathcal {A}\) with length \(l \ge L_{ \mathrm{min} }\) is selected from the all arcs in an image. The CCA hypotheses are generated using a RANSAC procedure modified from [13]. A hypothesis is generated from the arc if the hypothesis satisfies the following requirements: (i) it receives sufficient support from the the pixels located in the CCA area and (ii) the support comes from the sufficient number of directions.

The score for each hypothesis is computed as overall contribution of all edges to support the hypothesis

$$\begin{aligned} f_{h} =\frac{1}{R}\sum _{i=1}^{N}f_{p_{i}} \end{aligned}$$
(2)

where N is the number of supporting edges, R is the hypothetical radius of the CCA hypothesis and \(f_{p_{i}}\) is the support of an edge \(p_{i}\) computed using Eq. 1.

The number of directions from which a hypothesis gets support is determined to ensure the evidence of a bubble from multiple directions. The edges, voting for a hypothesis H with radius R, are divided into K sectors according to their orientation. The decision whether a hypothesis should be rejected is made based on the value \(N_{s}\), the number of sectors where the support of the hypothesis is not zero. The center and the radius of the hypothesis are optimized by maximizing the score function \(f_h\) with the Nedler–Mead Simplex method [21]. The optimization is used to capture the outer rim of the bubble by taking into account all the votes from the inner structure. The center location is then updated according to the radius.

Finally, the non-maximum suppression is performed on the CCA parameters to prevent multiple hypotheses for a single bubble. A hypothesis \(H_i\) with support \(f_i\) is rejected if there is a hypothesis \(H_j\) such that \(|r_{i}-r_{j}|<0.5r_{j}\), \(|\mathbf {c_{i}}-\mathbf {c_{j}}|<0.5r_{j}\), and the hypothesis \(h_j\) received greater support than the hypothesis \(h_i\).

4 Experiments and discussion

The CCA detection method was tested on two sets of data: (i) pulp suspension images to estimate the gas volume contained in the bubbles and (ii) oil dispersion images [10]. The performance of the CCA method, standard OpenCV HT and sliding window method with WaldBoost detector [24] were compared. The parameter selection was also considered.

Assuming that the bubbles have an approximately spherical shape, the volume of a bubble with radius R is \(V = \frac{4}{3}\pi R^3\). Since the primary objective was to estimate the gas volume, the volume estimation error was used as the performance criterion. The mean relative error of volume estimation for M images was computed as

$$\begin{aligned} V_{\mathrm{err}} = \frac{1}{M}\sum _{i=1}^{M}\left( \frac{|V_{est_{i}} - V_{gt_{i}}|}{V_{gt_{i}}}\right) , \end{aligned}$$
(3)

where \(V_{gt_{i}}\) is the ground truth volume and \(V_{est_{i}}\) is the estimated volume.

To present the evaluation results, we used the following notation: true positive (TP) means that a detected bubble is present in the ground truth, false positive (FP) that a detected bubble is not among the ground truth bubbles, and false negative (FN) that a bubble in the ground truth was not detected. A bubble was considered as correctly detected if the distance of the centers and the difference of radii of the detected bubble and the ground truth bubble were less than \(\nu = 35~\%\) of the radius of the ground truth bubble. A ground truth bubble was matched with at most one detected bubble. For the further analysis of parameter values and their effect on the accuracy of estimating the bubble size distribution, true-positive Rate defined as \(TPR = \frac{TP}{TP + FN}\) was used.

4.1 CCA performance on pulp suspension images

The dataset included 24 fully annotated microscopic images with the resolution of 1600\(\times \)1200 pixels, see Fig. 2. The images, provided by the FiberLaboratory, were obtained from a pilot mixing setup. In total, 1141 bubbles in the images were marked by an expert drawing the outer rim of the bubbles, and the ground truth volume was computed from the marked bubbles. The data were divided into a training set containing eight randomly selected images (397 bubbles) and a test set containing the rest of the images, 16 images (744 bubbles).

The following parameters were selected based on the training data: edge detection filter size of 19 pixels, the number of CCA sectors \(N = 10\), the required number of sectors \(N_s = 4\), the maximum radius of bubbles \(R_{\mathrm{max}} = 0.152~\mathrm {mm}\), the filter response threshold \(T_{ filt } = 0.93\), the width of the annulus \(\theta = 0.5R\), and the minimum length of an arc from which a hypothesis is generated \(L_{\mathrm{min}} = 15\). The parameter selection procedure was repeated 4 times and only minor variation in selected parameter values was observed [25]. The parameter of the cost function, \(w = 0.6R\) was obtained using ground truth annotations. It should be noted that small variation in the parameter value caused by imperfect ground truth markings does not have remarkable effect on the results as the method is not highly sensitive to the parameter [25].

A detailed parameter sensitivity analysis can be found in [25]. To estimate the error landscape, gas volume estimation error \(V_{\mathrm{err}}\) was computed fixing all parameters but two at a time and varying the two parameters. The error varied smoothly in some areas, but there were peaks corresponding to high error values. It could be explained by the fact that there is a big variation in the provided images and with certain parameter values in some of the images the error significantly increases. In addition, certain parameter combinations induced problems in detecting large bubbles causing big gas volume estimation error emphasizing the importance of the parameter selection step.

Fig. 5
figure 5

Detection of bubbles in pulp suspension images: a The number of bubbles as the function of size; b Bubble volume as the function of size

Full size image

The bubble counts and volumes summed over all the test images in five bubble size categories are shown in Fig. 5. In the dataset, most of the big bubbles with the radius about 0.16 mm (80 pixels) were detected correctly and there were no false positives among them. This is important since the gas contained in the bubbles is considerable and even one false negative in this group would cause a significant error in the volume estimation. The size range 0.11–0.14 mm (55–70 pixels) includes both FP detections and FNs. In the small bubble range (\(\le \)0.06 mm, 30 pixels), mostly the FNs are a problem.

In the intended application, the precision of the gas volume estimation is the key parameter. With the selected parameter settings, the mean relative error of volume estimation \(V_{err}\) for M images was 28 % (std 30 %) with the precision \({ TP}/({ TP}+{ FP}) = 74~\%\). We computed the precision value separately for the central parts of the images (676\(\times \)676): the result was 86 %. The relatively large difference can be explained by the fact that the peripheral parts of the images are out of focus as it can be seen in Fig. 2. This produces FP detections. Some bigger bubbles were not detected which caused higher volume estimation error.

The method works well for bubbles complying with the model, i.e., for bubbles with bright ridge edges as shown in Fig. 6. Small bubbles are often undetected because low image resolution alters their appearance as shown in Fig. 6. Technically, the problem could be solved by imaging with higher magnification. However, such small bubbles contain very little gas, and therefore, these FNs have a notable effect only on the estimation of the bubble size distribution. The presence of fibers in the images also causes FNs and FPs.

Fig. 6
figure 6

Examples of bubble detection by the CCA method. TP (blue), FN (red), FP (yellow)

Full size image

When the goal is to estimate the bubble size distribution more reliably, TPR should be used as the guideline for parameter selection. In the example with the three smaller groups of bubbles (radius smaller than 0.06 mm), we show how the the detection of small bubbles can be improved. We ran the method on randomly selected 8 images 4 times and for each size group, selected the parameters that provided the highest TPR. The experiment was run only for the groups with smaller bubbles since the number of bubbles in the other groups did not allow appropriate parameter value optimization per each size group. The average results of bubble detection on the test set is shown in Fig. 7. It is clear that depending on the application, the performance can be improved by optimizing the parameters.

The method was implemented in Matlab. Using a laptop with a 2.6 GHz CPU, the run time was on average 40 seconds per image. The computation time breakdown was as follows: orientation sensitive filtering 36 %, non-maximum suppression 4 %, thresholding and linking 4 %, hypothesis sampling 5 %, hypothesis optimization 50 %, hypothesis selection 1 %. The current run time is acceptable from a view of process control when the process takes several hours.

To see how the bubble hypothesis generation affects the results, we substituted our hypothesis generation stage with the standard OpenCV CHT, using the mexopencv library by [32]. The accumulator threshold for center detection was selected to 40 by threefold cross-validation on the training set. After the generation, the hypotheses were selected in the same way as in the original framework. The mean relative error of the volume estimation on the test set was 67 % with the precision of 75 %, therefore, providing slightly lower precision and considerably higher error of volume estimation compared to CCA.

Fig. 7
figure 7

Improvement in the detection rate of small bubbles when the method parameters are optimized by maximizing the true positive rate

Full size image

CCA method was also compared with the appearance-based sliding window approach with WaldBoost detector [24]. In terms of bubble detection the sliding window method demonstrated better results whereas the most precise volume estimate was produced by the CCA method. The detailed comparison of the two methods can be found in [26].

4.2 CCA performance on oil dispersion images

The second dataset was provided by Dominguez et al. [10] and included 236 images (all used as test set) with the resolution of 319\(\times \)240 containing 1114 oil drops marked by an expert. The method parameters were the same as in the pulp suspension experiments, but the maximum radius value was set to 30 as the maximum radius of the ground truth bubbles and the segmentation threshold was set to 0.7 selected based on the method performance. Examples of the detection results can be seen in Fig. 8.

Fig. 8
figure 8

Examples of oil dispersion images (top row) and examples of bubble detection by the CCA method (bottom row). TP (blue), FN (red), FP (yellow)

Full size image

The original work [10] focused on the detection of the oil drops only. The images contain also other circular objects such as air bubbles, but the provided GT contained only the parameters of the oil drops. Since the CCA method detects any circular objects, we were able to estimate only the TPR using the provided GT. For the purpose of estimating the volume, we calculated the accuracy of oil volume estimation. The following results were obtained: TPR related to the number of oil drops was 44 %, TPR related to the oil volume was 86 %, and the mean relative error of volume estimation computed using Eq. 3 was 13 %. Moreover, the total error of volume estimation error computed as \( V_{\mathrm{total~err}}= \frac{|V_{ \mathrm{est} } - V_{ \mathrm{gt} }|}{V_{ \mathrm{gt} }}\cdot 100~\%\) was only 3 %.

The OpenCV CHT [32] detected 18 % of the bubbles. However, TPR related to the oil volume was 53 % which is lower than that of the CCA method, and the mean relative error of volume estimation was 47 %.

5 Conclusions

We presented a method for the detection of bubbles, or more generally, transparent spherical objects in a liquid. The problem was formulated as the detection of Concentric Circular Arrangements (CCA) which are recovered in a hypothesize–optimize–verify framework. The proposed method for bubble detection showed satisfactory performance in an industrial application requiring the estimation of gas volume in pulp suspension, achieving the mean relative error of 28 % with the precision of 74 % for the whole images and 86 % for the central parts. The detection takes approximately 40 s per image using a moderate computer which is tolerable from the process control point-of-view. The generalizability of the method was demonstrated on the oil dispersion images. In the future work, more research will be done on the method acceleration and method development to eliminate false-positive detections.