# A user-friendly interactive framework for unsteady fluid flow segmentation and visualization

### Abstract

While vector fields are essential to simulate a large amount of natural phenomena, the difficulty to identify patterns and predict behaviors makes the visual segmentation in simulations an attractive and powerful tool. In this paper, we present a novel user-steered segmentation framework to cope with steady as well as unsteady vector fields on fluid flow simulations. Given a discrete vector field, our approach extracts multi-valued features from the field by exploiting its streamline structures so that these features are mapped to a visual space through a multidimensional projection technique. From an easy-to-handle interface, the user can interact with the projected data so as to partition and explore the most relevant vector features in a guidance frame of the simulation. Besides navigating and visually mining structures of interest, the interactivity with the projected data also allows the user to progressively enhance the segmentation result according to his insights. Finally, to successfully deal with unsteady simulations, the segments previously annotated by the user are used as a training set for a Support Vector Machine approach that classifies the remaining frames in the flow. We attest the effectiveness and versatility of our methodology throughout a set of classical physical-inspired applications on fluid flow simulations as depicted in the experiment results section.

### Graphical Abstract

## Keywords

Flow segmentation Time-varying visualization Vector field Interactive tools Machine learning## 1 Introduction

Segmenting vector fields is of key importance in the context of scientific illustration and visualization. The fluid mechanics community, for example, faces the problem when simulating the motion of a fluid, to capture different structures of the flow such as vortices, saddle points and skins (Post et al. 2002). Aiming at segmenting vector fields that commonly appear in computational simulations, three major categories have gained a great amount of attention in the last decade: topology-based, geometric-based, and feature space-based.

Methods that purely rely on topology extract critical points from the vector field so that the topological structures of these points are conveniently handled and classified to cluster different regions of the field (Mahrous et al. 2004; Laramee et al. 2007). Despite their robustness in fragmenting arbitrary vector fields (Bhatia et al. 2011, 2012), approaches based on topology are typically computationally expensive, being intricate also to be coded in most of the cases. Geometry-based methods profit from the geometric attributes of the vector field such as curvatures, gradients, and streamlines, where the vector flow is partitioned according to the similarity between geometric representatives of the output segments. Although intuitive and effective (Ferreira et al. 2013; Li et al. 2006; Kuhn et al. 2011; Zhang et al. 2016a, b), geometry-based methods often suffer from the absence of a user interactive interface to adequately customize the segmentation. Finally, the last class of methods takes a specified set of features computed from the vector field, so that regions of similar flow behavior are grouped and classified (Daniels et al. 2010; Rossl and Theisel 2012; Motta et al. 2015). In fact, feature space-based algorithms usually perform well when dealing with steady vector fields (a static frame extracted from the flow), however, processing time-dependent sequence of frames (the unsteady case) is still a hurdle for the majority of these methods.

In this work, we introduce a new technique for segmenting both steady and unsteady vector fields which combines streamlines, multidimensional projection, and supervised machine learning into an innovative and unified framework. In contrast to most existing algorithms, in particular, the three basic groups mentioned above that formally tackle the segmentation task as a well-behaved steady-state problem, our approach enables to segment multiple frames in a time-varying vector field, a challenging and more general case due to the preservation of the temporal coherence between the frames. Moreover, in our formulation, the user can conveniently drive the segmentation by freely interacting with projected data obtained from a feature space in a 2D visual interface, leading, therefore, to alternatives for customization issues. As we shall show, the clusters annotated by the user in a given frame of the vector flow are automatically propagated to the others by applying modern concepts of machine learning, preserving the user’s choice while still avoiding unnecessary side processing in all the remaining frames of the simulation.

## 2 Pipeline overview

*D*visual space, so as to preserve neighborhood structures between the streamlines of the vector field as much as possible. Clustering is then performed in the visual space according to the prior subdivision of partitions by the viewer, thus highlighting the geometric patterns that she/he intends to visualize in the simulation. Finally, in the last stage, we train a support vector machine (SVM) mechanism to automatically propagate the traits previously specified by the user to the remaining frames of the vector field. Another interactive aspect of our framework is that the viewer can also assign new clusters to be exploited once the simulation is being run (see the bottom arrow in Fig. 1).

As a boosting step for our approach, we run the feature extraction apparatus described in Motta et al. (2015) aiming at producing a set of high-dimensional points for each frame of the simulation. This gives rise to our multidimensional projection scheme, which is carried out by the LAMP algorithm (Joia et al. 2011). In the following, we briefly discuss the technical aspects regarding the flow features used to construct our multidimensional attribute space.

## 3 Features description and quantization

*S*obtained as in Motta et al. (2015), we extract a set of discriminative features representing different properties of the flow so that these collected attributes are quantized and prescribed as feature vectors into a high-dimensional space. These attributes are numerically computed from the vector field

*V*by employing basic Finite Difference (FD) rules and then interpolated for each streamline point \(s_i \in S\) through a regular bicubic (or tricubic, in the 3D case) interpolation scheme. So, let \({\mathbf {v}}\) be the local velocity at a certain streamline point \(s_i \in S\). We then consider the following attributes to represent our collection of differential and geometric flow features:

- 1.
*Magnitude of Velocity*(\(\Vert {\mathbf {v}}\Vert\)) determines the intensity of the resulting velocity in a certain point of the vector field*V*(Sternberg and Loomis 2014). - 2.
*Magnitude of Vorticity*(\(\Vert {\mathbf {\omega }}\Vert\)) measures how much a fluid portion locally rotates as it moves with its flow (Sternberg and Loomis 2014). - 3.
*Divergence*(\(\nabla \cdot {\mathbf {v}}\)) gauges how much a fluid portion is being transported away from the neighborhood of a given point located in the flow, in other words, it quantifies the decay rate of the density at this point (Sternberg and Loomis 2014). - 4.
*Curvature*(\(\kappa\)) can be intuitively understood as a measure of how sharply a curve is turning as it is traversed. In our implementation, we compute the curvature as suggested by McLoughlin et al. (2012):where \({\mathbf {a}}\) is the local acceleration, computed by \({\mathbf {J}} \cdot {\mathbf {v}}\), with \({\mathbf {J}}_{ij} = {\partial v_i}/{\partial x_j}\) being the coefficients of the Jacobian matrix.$$\begin{aligned} \kappa = \frac{\Vert {\mathbf {v}} \times {\mathbf {a}} \Vert }{\Vert {\mathbf {v}}\Vert ^3} \,, \end{aligned}$$(1) - 5.
*Torsion*(\(\tau\)) accounts for the twisting of a particle embedded within a 3D vector field. So, we include \(\tau\) as a valid attribute for 3D simulations, which is computed as follows (McLoughlin et al. 2012):$$\begin{aligned} \tau = \frac{({\mathbf {v}} \times {\mathbf {a}})\cdot ({\mathbf {J}} \cdot {\mathbf {a}})}{\Vert {\mathbf {v}} \times {\mathbf {a}} \Vert ^2} \,. \end{aligned}$$(2)

*S*, an attribute value \(A^k_i\) for each one of the above-mentioned features

*k*is designed to represent the attributes of \(s_i \in S\). These values are then distributed, by features, to form a frequency histogram of

*m*bins. Finally, a composite histogram is created by concatenating the feature histograms into a 4

*m*-dimensional attribute vector (see Fig. 2), in summary: a streamline

*S*is represented by a 4

*m*-dimensional point in a feature space. In our experiments we take \(m = 10\), which leads to high-dimensional spaces of 40 and 50 dimensions for 2D and 3D vector fields, respectively.

## 4 Multidimensional projection

In this step, we focus on mapping the high-dimensional instances obtained from the feature extraction process onto a two-dimensional visual space. For this purpose, we use the multidimensional projection technique called Local Affine Multidimensional Projection (LAMP) (Joia et al. 2011) to project the data as well as to assist the segmentation process, in an effort to discriminating patterns of interest into separated clusters. Besides preserving neighborhood structures efficiently during the projection step, the LAMP technique also enables a very flexible mechanism to interactively modify the projection according to the user instructions. Notice that this is an important trait in our context (which also covers time-varying vector fields), as the process of segmenting “frame-by-frame” the whole simulation tends to be laborious and very time-consuming. Furthermore, using this kind of interactive proposal, one could also avoid the preliminary task of segmenting completely a boosting simulation frame before propagating its segments to the other frames, which indeed leads to be more applicable and opportune for practical usages.

Once the high-dimensional instances have been projected, the mapped points are then used to support the next step of our framework as an effective and exploratory tool to interactively label and mine structures of interest in the vector field.

### 4.1 User manipulation of clusters

Besides being easy-to-operate and robust in terms of clustering similar streamline-coded points, our technique also enables an interactive mechanism which allows for repartitioning data by “dragging and drop” the projected control points. When a control point is moved in the interactive plane, the data in its neighborhood is moved together, thus allowing to control the neighborhood structures in the visual space. So, if the projection result is not satisfactory, the user can select badly clustered points, turning them into new control points that can be manually moved in the interactive layout to better refine clusters and, thereby, improve the streamlines classification. Figure 4 illustrates a user intervention case where control points are dragged to different clusters within the interactive layout.

### 4.2 Control points assignment

*N*be the number of streamline vectors in \({\mathbf {X}}\) taken from a given guidance frame \(F_j\), and \(n_{\mathrm{S}}\) be the number of control points in the projection plane \({\mathbf {Y}}\). As suggested in Joia et al. (2011), we set \(n_{\mathrm{S}} = \left\lceil \sqrt{N}\right\rceil\) as a prior estimation to pave the way for reaching a good level of accuracy in our opening multidimensional mapping. So, as a useful and predictive scheme to get the best candidates for the control points, we first subdivide the vector field domain into a

*d*-dimensional regular grid \({\mathbf {G}}\) containing \(\left\lceil \root d \of {n_{\mathrm{S}}}\right\rceil\) cells in each axis, wherein

*d*stands for the vector field dimension. For each cell \(C \in {\mathbf {G}}\), let \({\mathbf {F}}_C=\{{\mathbf {\phi }}_i\}_{i=1}^N\) be the set of feature vectors corresponding the streamlines which intersect

*C*. The control point associated to the cell

*C*is the feature vector \(\phi _j \in {\mathbf {F}}_C\) which minimizes the following expression:

## 5 Learning segmentation on unsteady vector fields

Let \(\tilde{\mathbf {X}}\) be the multidimensional feature space formed by all streamline-like points taken from the frames \(F_i\) over time, \(i=1,2,\ldots ,n\). We start from the projected 2*D* points grouped as multiple clusters by the user in the guidance frame \(F_j\) managed as in the previous step of our framework. So, clusters as well as the high-dimensional data \(\tilde{\mathbf {X}}\) are used to drive the streamlines classification in all time-dependent frames \(F_i\), \(i \ne j\). More precisely, we create a training set for a machine learning method which learns and sorts out the feature points of the frames according to their level of similarity with the features of \(F_j\). To accomplish this task, we employ a multi-class version of the *Support Vector Machine* (SVM) (Dietterich and Bakiri 1995) on \({\tilde{\mathbf {X}}}\).

We apply the SVM to construct a training set \({\mathbf {Q}}\) which reflects the user-specified traits from the guidance frame \(F_j\). This training set is used to generate a set of hyperplanes \(\tilde{\mathbf {H}}\) in \(\tilde{\mathbf {X}}\), so that \(\tilde{\mathbf {H}}\) splits \(\tilde{\mathbf {X}}\) into multiple disjoint subspaces. As a result, the sampled points from \(\tilde{\mathbf {X}}\) are classified, so that each streamline located at \(F_i\), \(i \ne j\), is labeled in terms of a cluster learned in \(F_j\). In practice, to reduce the influence of outliers on \(\tilde{\mathbf {X}}\), we also introduce in the classification process the lower and upper bounds of histograms encoded by the streamlines. Such a procedure is performed by imposing for each set of multi-featured frame points, the global average of the histogram bounds as they range all the unclassified frames.

The strength and versatility of the machine learning theory smooth the path to properly manage a large amount of frames while achieving multiple segmentations. Indeed, this learning scheme turns the segmentation task simpler and less laborious on unsteady vector fields, avoiding the exhaustive and individual supervision of the frames by the user. Furthermore, regions with similar streamlines in the vector field can be easily identified from the projected data in the visual space, pointing out where users should nail down their exploration to propagate to the other frames the patterns previously labeled in \(F_j\).

## 6 Experiments

To demonstrate the robustness and flexibility of our approach to segment arbitrary vector fields, we compare it against two state-of-the-art visualization methods namely Daniels’ method (Daniels et al. 2010) and the Edge maps tool (Bhatia et al. 2012). We also evaluate our framework in a set of challenging cases comprising time-varying vector fields (the unsteady case), as the ones conducted by classical simulations of computational fluid dynamics (CFD) such as Rayleigh–Taylor effect and von Kármán instabilities. So, we focus on properly classifying the vector fields into multiple clusters so that they can be seen by the user as a junction of different flow patterns (e.g., rotational, curvilinear, laminar, turbulent flow, etc).

To computationally create our numerical simulations, we employ the well-known Gerris solver (Popinet 2003). Considering the unsteady set of experiments, we take a time step \(\delta _t = 0.05\) for the streamline integration and we assign 10 for the number of bins of the frequency histograms (see Sec. 3). Finally, as described before, we make use of the following geometric features to create multi-valued vectors to be used in our classification scheme: curvature, divergence, magnitude of velocity/vorticity, and torsion (for 3D vector fields, only).

### 6.1 The steady case

Figure 8 demonstrates the good performance of the proposed methodology when evaluated against the well-established Edge Maps method (Bhatia et al. 2012), a powerful state-of-the-art technique which takes advantage of topological properties of the vector field to accomplish the clustering. One can verify from the results depicted in Fig. 8 that our approach originates quite similar partitions to the ones coming from the Edge Maps, attesting the positive performance of the designed technique even though against a robust (although a labor-intensive tool to be properly coded and tuned) visualization tool such as Edge Maps method.

### 6.2 The unsteady case

We now concentrate our experimental analysis towards dealing with the harder case of unsteady fluid flow simulations, that is, when the flow varies as the time increases. For this purpose, we consider three simulations as reported below.

^{1}

Steady and unsteady vector fields characteristics and timings (in seconds)

## 7 Conclusion

In this paper, we address the problem of segmenting steady as well as unsteady fluid flow simulations as an interactive framework that unifies multidimensional projection, machine learning concepts and the user capability in grouping data into different clusters of interest. Despite its simplicity, flexibility and robustness in segmenting vector fields, our approach can succeed in many scenarios, ranging from artificial to “real world” collected data, as verified by the experimental results. Moreover, the proposed methodology achieves very pleasant results when compared to existing representative techniques and also providing useful traits as interactivity flexibility and easiness of use. In summary, all these characteristics render the proposed framework a very attractive interactive tool in the context of fluid flow visualization, specially to handle the complex case of unsteady simulations.

As future work, we intend to incorporate new features extracted from the vector field into our framework, such as the energy of the flow provided by the POD (*Proper Orthogonal Decomposition*) (Berkooz et al. 1993) modes. Another direction is to port our method to GPU architecture, since the classification of unsteady vector fields can also be accomplished in parallel.

## Footnotes

- 1.
Check the complete video: http://icmc.usp.br/e/13837.

## Notes

### Acknowledgements

We would like to thank Harsh Bhatia to kindly provide the results with Edge Maps (Bhatia et al. 2012). This research has been supported by FAPESP (São Paulo Research Foundation: grants #2013/07375-0; #2014/16857-0), and CAPES (PDSE #133553/2016-01).

## References

- Berkooz G, Holmes P, Lumley JL (1993) The proper orthogonal decomposition in the analysis of turbulent flows. Annu Rev Fluid Mech 25:539–575MathSciNetCrossRefGoogle Scholar
- Bhatia H, Jadhav S, Pascucci V, Chen G, Levine JA, Nonato LG, Bremer PT (2011) Edge maps: representing flow with bounded error. IEEE Pacific Vis 2011:75–82Google Scholar
- Bhatia H, Jadhav S, Pascucci V, Chen G, Levine JA, Nonato LG, Bremer PT (2012) Flow visualization with quantified spatial and temporal errors using edge maps. IEEE Trans Vis Comput Graph 18(9):1383–1396CrossRefGoogle Scholar
- Daniels J II, Anderson E, Nonato L, Silva C (2010) Interactive vector field feature identification. IEEE Trans Vis Comput Graph 16(6):1560–1568CrossRefGoogle Scholar
- Dietterich TG, Bakiri G (1995) Solving multiclass learning problems via error-correcting output codes. J Artif Intell Res 2:263–286zbMATHGoogle Scholar
- Ferreira N, Klosowski J, Scheidegger C, Silva C (2013) Vector field k-means: clustering trajectories by fitting multiple vector fields. Comput Graph Forum 32:201–210CrossRefGoogle Scholar
- von Funck W, Weinkauf T, Theisel H, Seidel HP (2008) Smoke surfaces: An interactive flow visualization technique inspired by real-world flow experiments. IEEE Trans Vis Comput Graph 14(6):1396–1403CrossRefGoogle Scholar
- Garcia D (2010) Robust smoothing of gridded data in one and higher dimensions with missing values. Comput Stat Data Anal 54(4):1167–1178MathSciNetCrossRefzbMATHGoogle Scholar
- Joia P, Coimbra D, Cuminato JA, Paulovich FV, Nonato LG (2011) Local affine multidimensional projection. IEEE Trans Vis Comput Graph 17(12):2563–2571CrossRefGoogle Scholar
- Kuhn A, Lehmann DJ, Gaststeiger R, Neugebauer M, Preim B, Theisel H (2011) A clustering-based visualization technique to emphasize meaningful regions of vector fields. Vis Model Vis 2011:191–198Google Scholar
- Laramee R, Hauser H, Zhao L, Post F (2007) Topology-based flow visualization, the state of the art. In: Topology-based methods in visualization, mathematics and visualization, pp 1–19. Springer (2007)Google Scholar
- Li H, Chen W, Shen IF (2006) Segmentation of discrete vector fields. IEEE Trans Vis Comput Graph 12(3):289–300CrossRefGoogle Scholar
- Mahrous K, Bennett J, Scheuermann G, Hamann B, Joy KI (2004) Topological segmentation in three-dimensional vector fields. IEEE Trans Vis Comput Graph 10(2):198–205CrossRefGoogle Scholar
- McLoughlin T, Jones MW, Laramee RS, Malki R, Masters I, Hansen CD (2012) Similarity measures for enhancing interactive streamline seeding. IEEE Trans Vis Comput Graph 99:1–1Google Scholar
- Motta D, Oliveira M, Pagliosa P, Nonato LG, Paiva A (2015) Exploratory segmentation of vector fields using multidimensional projection. In: Sibgrapi 2015 (28th conference on graphics, patterns and images), pp 250–256Google Scholar
- Popinet S (2003) Gerris: a tree-based adaptive solver for the incompressible Euler equations in complex geometries. J Comput Phys 190(2):572–600MathSciNetCrossRefzbMATHGoogle Scholar
- Post FH, Vrolijk B, Hauser H, Laramee RS, Doleisch H (2002) Feature extraction and visualization of flow fields. In: Eurographics 2002 STAR, pp 69–100Google Scholar
- Rossl C, Theisel H (2012) Streamline embedding for 3D vector field exploration. IEEE Trans Vis Comput Graph 18(3):407–420CrossRefGoogle Scholar
- Sternberg SZ, Loomis LH (2014) Advanced calculus, revised edn. World Scientific Publishing Company, SingaporezbMATHGoogle Scholar
- Zhang L, Chen G, Laramee RS, Thompson D, Sescu A (2016) Flow visualization based on a derived rotation field. Electron Imaging 2016(1):1–10Google Scholar
- Zhang L, Laramee RS, Thompson D, Sescu A, Chen G (2016b) An integral curve attribute based flow segmentation. J Vis 19(3):423–436CrossRefGoogle Scholar