The Multi-strand Graph for a PTZ Tracker

Abstract

High-resolution images can be used to resolve matching ambiguities between trajectory fragments (tracklets), which is a key challenge in multiple-target tracking. A pan–tilt–zoom (PTZ) camera, which can pan, tilt and zoom, is a powerful and efficient tool that offers both close-up views and wide area coverage on demand. The wide area enables tracking of many targets, while the close-up view allows individuals to be identified from high-resolution images of their faces. A central component of a PTZ tracking system is a scheduling algorithm that determines which target to zoom in on, particularly when the high-resolution images are also used for tracklet matching. In this paper, we study this scheduling problem from a theoretical perspective. We propose a novel data structure, the Multi-strand Tracking Graph (MSG), which represents the set of tracklets computed by a tracker and the possible associations between them. The MSG allows efficient scheduling as well as resolving of matching ambiguities between tracklets. The main feature of the MSG is the auxiliary data saved in each vertex, which allows efficient computation while avoiding time-consuming graph traversal. Synthetic data simulations are used to evaluate our scheduling algorithm and to demonstrate its superiority over a naïve one.

Appendix A: Labeling Extensions

In this appendix we describe a refined computation of $r_1(v)$ (Sect. 4.2.1). The labeled tracklet of each labeled origin of v, $u\in O^\star (v)$, clearly consists of the tracklet represented by u itself. In addition, a labeling of a vertex can sometimes also be extended to its descendants. Let u be a labeled vertex with a single compound child, $u_\mathrm{c}$ (for example, see the labeled $v_1$ and its compound child $v_4$ in Fig. 4a). The tracklet of $u_\mathrm{c}$ follows the tracklet of u, and both clearly represent the same target. Hence, the labeled tracklet of u can be extended to include also the tracklet of $u_\mathrm{c}$. As a result, the length of the labeled tracklet is given by $\tau (u)+\tau (u_\mathrm{c})$. Additional vertices may also be added to this length, as follows. We define the forward labeling extension to be the path $\ell (u_\mathrm{c},u')$ in which each vertex is a single compound child of its parent. The computation of $r_1(v)$ can be refined by considering forward labeling extensions of the labeled origins of v, as described next.

Let us consider again the path $\ell (u,v)$, where u is a labeled origin of v. We wish to find the contribution of u to $r_1(v)$ when considering not only u itself but also its forward labeling extension. This contribution excludes the entire extension of u, which is labeled prior to the labeling of v. The length of the forward labeling extension of u, $\tau (\ell (u_\mathrm{c},u'))$, is therefore subtracted from $\tau (\ell (u,v))$. That is, the contribution of the possible matching of u to v is given by $\tau (\ell (u,v))-\tau (\ell (u_\mathrm{c},u'))$. We next describe the auxiliary data needed to compute the refined $r_1(v)$ efficiently.

For each vertex v, we define $n_\mathrm{ret}(v)$ to be the number of forward labeling extensions of the labeled origins of v in which v is included. This value can be recursively computed based only on the vertex itself and its direct parents, as follows:

$$\begin{aligned} n_\mathrm{ret}(v)=\left\{ \begin{array}{l@{\quad }l} \sum \limits _{v_i\in P(v)}n_\mathrm{ret}(v_i)\cdot {\psi }(v_i) &{} v\notin V^\star \\ 1 &{} v\in V^\star ,\\ \end{array} \right. \end{aligned}$$

(22)

where ${\psi }(v)$ is a binary value that determines whether v has only one compound child. Note that $n_\mathrm{ret}(v)\le n_o^\star (v)$ and that $n_\mathrm{ret}$ equals 1 for a labeled origin and 0 for an unlabeled origin.

The refined recursive computation of $r_1(v)$ (that replaces Eq. 14 above) is given by:

$$\begin{aligned} r_1(v)=\left\{ \begin{array}{l@{\quad }l} \tau (v)\cdot (n_o^\star (v)-n_\mathrm{ret}(v))+\sum \limits _{v_i\in P(v)}r_1(v_i) &{} v\notin V^\star \\ 0 &{} v\in V^\star .\\ \end{array} \right. \end{aligned}$$

(23)

For example, in Fig. 4a we wish to compute $r_1(v_9)$, that sums the contributions of $v_1$ and $v_3$, the two labeled origins of the unlabeled vertex $v_9$. Due to forward labeling extensions, $v_1$ contributes $\tau (\ell (v_5,v_9))$ and $v_3$ contributes $\tau (v_9)$.

The labeling of a vertex can also be extended backwards, in a manner similar to the forward labeling extension (as explained in Sect. 4.2.1). While the refined $r_1(v)$ computation uses only the forward labeling extension, the $r_2(v)$ computation in Eq. 20 uses only the backward labeling extension. For example, in Fig. 5b, only $v_3$ is labeled. The unlabeled vertex $v_9$ has two unlabeled origins, $v_1$ and $v_2$, and a potential match by elimination, $v_5$. We wish to compute $r_2(v_9)$, that sums the expected increase of $L^\star$ (Eq. 10) over $v_1$ and $v_2$, in the event where $v_9$ is labeled and is matched to $v_5$. Since $v_5$ has a single compound parent, $v_4$, the tracklet of $v_5$ can be backward extended to include also the tracklet of $v_4$. Indeed, the recursive computation (Eq. 20) results in $r_2(v_9)=2\tau (\ell (v_4,v_9))$.

Note that both forward and backward labeling extensions are considered in the experiments for the evaluation of our scheduler.