1 Introduction

In a video surveillance system it is not possible to observe all areas in each part of a building. Most often the observed places are: doors, entrances, exits, corridors and staircases. In these kinds of places, people are usually in motion. Accordingly, they appear, pass by, and leave the FOV (Field Of Vision) of a given camera. Spatio-temporal dependencies between cameras can be described with a directed (mostly bidirectional) weighted graph, the so-called topology graph. The decision which has to be made concerns determining if two visible objects seen by two different cameras represent the same physical object. Such decisions can be made based on following premises:

  • Two appearances of the same physical object should have significant resemblance. The resemblance can be measured by comparing visual features of the objects present in the view of the camera.

  • Certain paths of the object’s movement are more probable than others. This information is encoded in the topology graph. Therefore, the probability of the object appearing in a certain area can be numerically expressed.

  • Moreover, time dependencies are contained in the topology graph as well. The time period between the object observations is also taken into consideration.

  • The last premise is the behaviour of the object. It is frequently observed that certain objects follow typical and repeatable paths. The knowledge of their typical behaviour should be gathered and stored in the system to support decision making.

The first three clues can be obtained instantly when two observations are available. However, in order to acquire knowledge of the behaviour of objects many more observations are necessary. Moreover, the behaviour can change over time, therefore a reaction to these fluctuations is desired. The gathered knowledge should be stored in a form which makes it easily accessible. The idea of Pawlak’s flowgraph (hereinafter referred to as FG) complies with these conditions. Prediction of the next steps of the object’s movement is expressed in the form of a probability tree. Every time an object is observed in the supervised area, its path is added to the set of observed paths. Subsequently, required changes are applied to the FG, incorporating the newly gathered knowledge. In order to keep the FG up to date, the paths stored in the system only cover a specified period of time (e.g., one month).

The remaining part of paper is organized as follows. In Section 2, the related work on using FGs as a tool for decision making is discussed. Methods for adapting FGs to changing trends in the analyzed environment are also presented. Additionally, other methods for prediction of object movement are mentioned. Section 3 describes the details of employing FGs for route reconstruction. We introduce, how the knowledge is gathered from past observations to construct the FG. Next, the process of refreshing the parameters of the FG is explained. Finally, the algorithm of movement prediction is mentioned. In Section 4, we use the proposed methods with the example data in order to clearly present to the reader the details of the engineered method. Section 5 describes computation complexity issues related to FGs. Section 6 contains the results and discussion of using the proposed algorithm with real data from a video surveillance system. Finally, Section 7 presents our conclusions and plans for future development of the method are discussed.

2 Related work

The foundations of the flowgraph idea were provided by Greco et al. (2002) and Pawlak (2005b). Many extensions to the original FG concept have been made. Simplicity of the FG and its applicability to problems of decision making and prediction are significant advantages. Determining strength, certainty and coverage values for paths in the FG has also been described in the literature (Pawlak 2005b). In related publications it has been shown how the FG can be used to create a decision tree (Pawlak 2006b). Moreover, the idea of FGs is strongly connected with Rough Sets and the Bayes’ theorem (Pawlak 2003). Clear rules that control the FG allow for quick understanding of its principles, as well as for adaptation of this idea in numerous fields, including data mining (Pawlak 2005a), music metadata processing (Czyzewski and Kostek 2004), and conflict analysis and negotiations (Pawlak 2005c, 2006a). The FG is even applicable to making decision about deregulation of aviation market. Such solutions have been introduced, for instance, by the government of Taiwan (Ou Yang et al. 2008).

In order to predict the movement of an object between non-overlapping cameras, methods such as particle filters, joint probabilistic data association, Markov chains, Bayesian networks and other statistical algorithms can be used. Lev-Tov and Moses (2010) assign objects to two types of locations: unobserved (hidden) or visible to the camera. The state of the given object is composed of a vector of appearance descriptors and locations (hidden or visible) which the object is currently observed in. In order to decide in which state the object is, a particle filter is applied. Cheng et al. (2006) used joint probabilistic data association of visual features for matching of tracked objects. Javed et al. (2003) proposed a method for fusion of many types of clues for matching the same object in multiple non-overlapped cameras. A framework based on the Bayes Theorem was applied for this purpose. Kim et al. (2009) proposed an approach to the problem of tracking people utilizing a method based on Markov Chain Monte Carlo (MCMC). In this method probabilities of transitions between cameras are given a-priori. Moreover, Leung et al. (2008) proposed a probabilistic method based on a probability density function (p.d.f.) with entrance and exit events treated as random events. Additionally, determination of the probabilities of the object’s possible routes is achieved with a dispersion function which can be interpreted as the conditional probability of a given exit event when a particular entrance event has occurred.

Movement of people through the supervised area can be considered as a flow of objects. Thus, such a phenomenon can be described and analyzed with the FG and methods employed for this task are described in the following subsections.

3 Proposed methods

In a video surveillance system in which the algorithms for making paths of the objects are implemented, methods for Forward Route Reconstruction can be applied as well. If the surveillance system has been operating for some time, certain statistics of the object movement can be gathered. Collected input data for Forward Route Reconstruction are the set of paths of observed objects from the past, as well as new, recently finished paths. The outcome encompasses the probability of object next path, provided it remains in the supervised area.

3.1 Building the flowgraph

In order to use Pawlak’s flowgraph for prediction, some assumptions need to be made. A path is constructed from observations of an object in subsequent cameras. An observation is expressed by the identifier of a camera and the index determining which step of the path this observation denotes. Thus, these two parameters, as a pair, can be considered as an attribute for the FG. Furthermore, a class of attributes for the FG can be defined as a subset of all attributes which have the same index of step. It means that each consecutive vertical layer of vertices in the FG corresponds to the subsequent step in the path.

According to work of Greco et al. (2002), an information system S = (U, A) can be described on the basis of a set of paths. Thus, U is a set of objects that enter and exit the area observed by the surveillance system and A is a set of the abovementioned attributes. Transforming the set of paths to FG is done using the following formulae. In (1), σ(x i , y j ) defines the rate of objects passing from camera x in step i of the path to camera y in step j of the path. The number of objects passing from x i to y j is denoted as φ(x i , y j ), and the total number of paths in the database is denoted as φ(G).

$$ \sigma\left(x_i, y_j\right) = \frac{\varphi\left(x_i, y_j\right)}{\varphi\left(G\right)} \label{eq:1} $$
(1)

Similarly, in (2), φ(x i ) is the number of objects that passed the node x i .

$$ \sigma\left(x_i\right) = \frac{\varphi\left(x_i\right)}{\varphi\left(G\right)} \label{eq:2} $$
(2)

The next two formulas ((3) and (4)) are used for probability estimation. The names of these parameters, that is certainty (cer) and coverage (cov), are taken from terminology related to FGs.

$$ cer\left(x_i, y_j\right) = \frac{\sigma\left(x_i, y_j\right)}{\sigma\left(x_i\right)} \label{eq:3} $$
(3)

The value of cer(x i , y j ) estimates the probability that the object which left camera x in step i of the path will appear in camera y in step j. Equation (4) is used to calculate the coverage factor cov(x i , y j ) for node y j , so that it determines estimation of probability that an object which appears in camera y in step j of the path was seen before in camera x in step i of the path.

$$ cov\left(x_i, y_j\right) = \frac{\sigma\left(x_i, y_j\right)}{\sigma\left(y_j\right)} \label{eq:4} $$
(4)

In (1), (3) and (4) j is equal to i + 1, thus it is related to the next step on the object path (the following layer in the FG).

The idea of FG needs some modification and adjustment in order to be used in Forward Route Reconstruction. The flow of objects through the video surveillance system can be described by a properly prepared FG. In such a system, the cameras which observe the entrances and exits (e.g., doors, gates) correspond to the first and last step of the path. The path of the object comprises several steps. Each step is associated to the object’s appearance in one of the cameras. The first step is easily obtained when the object is first observed in the observed area. Subsequent steps are derived from following observations. The last step, which denotes the object’s disappearance, is appended to the path after it has been observed that the object has left the supervised area. The FG encompasses all the observed paths. The nodes of the graph relate to steps on the object’s path. The FG is arranged in layers, whose number corresponds to the order of the particular nodes in the observed paths. Therefore, the first layer of the FG represents the entrance (input) nodes, whereas the exit (output) nodes can be in all layers but the first one. Additionally, the last layer of the FG contains only one exit node. In other words, since the lengths of paths can vary, exit nodes can appear in layers l ≥ 2. An example FG is presented in Fig. 1.

Fig. 1
figure 1

The simple example of a modified FG employed in Forward Route Reconstruction

Full size image

The example in Fig. 1 presents the FG for the monitoring system composed of two cameras (x and y) which can be both entered and left and transitions between them are possible in both directions. The longest considered path has the length of two steps. Additional output steps are marked as nodes o2 and o3.

3.2 Prediction of the object movement based on the flowgraph

On the basis of collected statistical data about the paths and employing (1)–(4), the FG is constructed and appropriate weights are assigned to the edges. The next step is using the prepared FG to predict the movement of an object located within the supervised area. The result of Forward Route Reconstruction is a probability tree describing probable next steps of objects. The root of the probability tree (denoted x root) is the node in FG which represents current step of the path of theobject which remains in the supervised area. The certainty values are used as probability measurements. In case of prediction more than one step ahead from the root, the probability needs to be estimated as a product of probabilities of subsequent steps. In (5) it is shown how the probability of the path from the node x root to z end is calculated (Pawlak 2005b):

$$ cer\left[x_{\rm root},\ldots,z_{\rm end}\right] = \sum\limits^{i=\mathrm{end}}_{i=\mathrm{root}} cer\left(x_i, y_{i+1}\right) \label{eq:5} $$
(5)

The x root position is changing as the object moves in the graph. Thus, the first node of the previous tree becomes a new root. Such prediction can be made as long as the object is present in the supervised area.

3.3 Updating the flowgraph

When the object enters the supervised area, it is assigned a copy of the FG. This copy is derived from the most up-to-date information. It is then used for movement prediction. The object’s copy of the FG remains unchanged as long as the object is present in the system. When the object leaves the area, its path is used to update the general FG.

Updating the FG is based on adding a new path to the set of observed paths. The updated set of paths is an input for refreshing the FG by using (1)–(4). Another method cleans up the set of paths in order to remove paths which are obsolete. This method is triggered periodically in order to keep the database of paths up to date and to enable adaptation of the FG to the changing behaviour of objects. Adjusting this adaptation period influences the rate of ‘forgetting’ information from the past.

4 Usage example of flowgraph

The prototype of an example FG for Forward Route Reconstruction is a topological structure of the considered video surveillance system described with the topology graph. In our assumed situation, there will be four cameras and two of them have entrance and exit in their FOV(nodes x and y). A detailed topology is shown in Fig. 2.

Fig. 2
figure 2

An example topology graph

Full size image

The next stage is to obtain statistical data. An example set of paths is presented in Table 1. Paths can have various lengths and the number of layers in the FG depends on paths lengths. The paths in Table 1 are used to construct the FG shown in Fig. 3. The number of layers equals the number of steps in the longest path plus one, because of adding exit (output) nodes. Analysis of the FG structure allows to reconstruct the topology graph on the basis of possible transitions between the cameras contained in the FG. In the graph presented in Fig. 3, some nodes are neither sources nor sinks, i.e., they are not connected to other nodes (e.g., w4, w5, z5). In this case camera was never a 4-th or 5-th step of any path. These nodes can be omitted and ignored in the process of creating the probability trees.

Table 1 An example set of paths
Full size table
Fig. 3
figure 3

The flowgraph constructed on the basis of a set of paths

Full size image

Equations (1)–(4) allow us to calculate the parameters for each edge in the FG. First, the number of transitions through each edge and through each node in the FG must be obtained. The measure of object flows through particular nodes is calculated according to (2) and presented in Table 2. Equation (1) determines the flows through the given edges. Therefore, a probability estimate of a particular transition can be obtained with (3) and (4). The listing of these probability measures is presented in Table 3.

Table 2 Flow measures of objects passing through given nodes
Full size table
Table 3 Flow and probability measures for edges in the flowgraph
Full size table

The FG prepared in such way can be used to predict the movement of an object O which is located within the observed area. The current path of the object O is, for example, x →z →?. Thus, in the FG, this object is localized in the node z2. The node z2 becomes the root of the nascent probability tree. The probability tree can be considered a subgraph of the FG, as it is shown in Fig. 4. In case of prediction of only one future step in the path of the object O, probability measures can be taken directly from the FG as values of cer(z2, x3), cer(z2, y3) and cer(z2, w3). If more than one-step prediction is needed, (5) can be used to obtain the probability tree with root in z2. In the example, the query [z2, w3, x4] results in cer[z2, w3, x4] = 0.3333 · 0.5000 = 0.1667. Example queries, which are rooted in the node z2, and responses to them are presented in Table 4.

Fig. 4
figure 4

Probability tree for the object O

Full size image
Table 4 Probability measures for the tree with the root in node z2
Full size table

In this example, when the object O exits the supervised area, its path, i.e. x →z →w →z →y, is added to the set of paths and the FG is refreshed and rebuilt. The new version of the FG is built on the updated set of paths. If the path of the object O is the same as the one present in the set of paths, there are no changes in the structure of the FG, but the values φ(x i , y j ), φ(x i ), φ(G), σ(x i , y j ), cer(x i , y j ), and cov(x i , y j ) are updated. The new FG parameters are presented in Tables 5 and 6. The values which must be updated because of adding a new path to the set of paths, are shaded. If another object enters the supervised area, it will get the newest updated version of the FG.

Table 5 Updated flow measures of objects passing through given nodes
Full size table
Table 6 Updated flow and probability measures for edges in the flowgraph
Full size table

5 Computational complexity

In order to determine computational complexity some assumptions must be made. These assumptions are related to a practical use of the flowgraph idea. A computer representation of the graph has an impact on the space complexity and the time complexity. In general, a graph can be stored in the computer memory as an adjacency matrix or as an adjacency list. Moreover, the matrix adjacency structure is more suitable for a dense graph that is a graph in which the number of edges is bigger than the squared number of vertices. The structure of the adjacency list is recommended for sparse graphs (Kubale 1998).

The following assumptions were made in order to determine the computational complexity: the number of cameras in system (corresponding to the number of vertices in a single layer) C is constant, the number of steps in the longest path (corresponding to the number of layers in the FG) L is equal to C. From the practical point of view the number of cameras in a deployed surveillance system changes very rarely. Additionally, we assume that all paths are as long as possible (the length is equal to L).

Thus, the number of vertices in the theoretically worse case can be estimated as V = L·(C + 1) ≈ C2. Therefore, the number of edges in the FG is strongly related to the number of edges in the topology graph. In the case, when a topology graph is a clique and the FG contains all possible transitions between consecutive layers, the maximum number of edges can be determined as E = C·[1 + (L − 1)·(C + 1)]) ≈ C3. Going one step further the FG can be considered as an L-partite graph. In this case, an application of the representation of adjacency list is more suitable for the FG.

The space complexity (amount of needed memory) of the FG represented as an adjacency list is equal to O(V + E). Following the assumptions that C and L are constant, the space complexity is O(1).

Application of associative containers to store vertices and edges entails a constant time of access or adding a vertex and O(log(E/V)) time complexity while accessing or adding an edge. The time complexity of the operation of adding a new sample (path) to the FG is O(L·log(E/V)) but according to the assumptions made, the time complexity is O(1). After adding a path to the FG (changing values of ϕ( )) an updating process (changing values of σ(), cer(), and cov()) occurs. This process has the time complexity O(C 2·log(E/V)). If we assume that C and L are constant, then the time complexity of updating process is O(1).

6 Results of experiments

The topology graph, on which the possible inter-camera transitions are presented, is shown in Fig. 5. For the experiment, four cameras were selected. Each camera was considered an entrance-exit camera. The views from these cameras are presented in Fig. 6.

Fig. 5
figure 5

The topology graph of the set of cameras in the experiment

Full size image
Fig. 6
figure 6

Screenshots of cameras fields of view used in the experiment, areas where people go in and go out from FOVs of the cameras are marked with yellow color

Full size image

In the presented setup, analysis of 6 hours of the recorded video was performed and 925 paths of objects were obtained. The analysis was made by a person but automated techniques application is also possible, provided that tracking methods are adequately efficient. Otherwise an error propagation would occur. The 925 paths are summarized in the FG (Fig. 7).

Fig. 7
figure 7

The flowgraph presenting the statistical structure of people behaviour

Full size image

The values of strength and certainty for each edge in the FG are contained in Table 7. Values of cer(x i , y j ) are used to create probability trees using the methods presented in previous sections. Probabilities of particular paths, which can be predicted from the probability tree with the root node placed in node A1 are presented in Table 8.

Table 7 Probability measurements calculated for the collected set of paths
Full size table
Table 8 Forward Route Reconstruction results—probabilities of future path of the object which is currently in the second step of its path and is observed by the Camera A
Full size table

In order to validate the knowledge contained in the prepared FG a set of the new paths gathered in the same place at different times were added to the previously created database of paths (for this purpose 101 new paths were obtained). Average absolute deviation is used as a measure of the new FG conformity with the old FG. Calculation of this measure is performed according to (6).

$$ D=\frac{\sum\limits_{\rm Edges}\left|cer_1\left(x_i,y_j\right)-cer_0\left(x_i,y_j\right)\right|}{N} \label{eq:6} $$
(6)

where D determines how much the new FG differs from the previous one, cer 1(x i , y j ) and cer 0(x i , y j ) are certainty parameters assigned to the edge x i →y j in the new and old FG respectively and N is the number of edges in the new FG. In the case when a new edge is added, which was not present in the old FG, we assume that cer 0(x i , y j ) = 0. The values of cer(x i , y j ) are normalized to 1, thus the value of D describes the degree of changes in the FG. When D is equal to 0 it is understood that no changes were made to the FG. Calculation of measure of conformity for the extended set of paths yields D = 0.0158. This value equals the average change in certainty of edges in the FG.

7 Conclusions

Pawlak’s flowgraphs can be successfully applied to video analysis in a surveillance system which observes areas inside buildings. The useful feature of the FG is automatic creation and updating of rules, as well as clear representation of the object route statistics. Moreover, probability measurements can be used to predict next steps of the object’s movement on the basis of a hitherto traversed path and information contained in the FG. The next advantage is possibility of using finished paths for updating the FG. Moreover, prediction of future steps in the path of an object can be made for various numbers of steps ahead of the current location.

An extension of the presented algorithm can be a more sophisticated method for updating the FG that would be based on assessment of changes in general behaviour of objects observed by a surveillance system, e.g., based on amount of information contained in FG. Additionally, representation of time-dependencies between cameras can also be included in the FG.