Complex Activity Recognition Via Attribute Dynamics

Abstract

The problem of modeling the dynamic structure of human activities is considered. Video is mapped to a semantic feature space, which encodes activity attribute probabilities over time. The binary dynamic system (BDS) model is proposed to jointly learn the distribution and dynamics of activities in this space. This is a non-linear dynamic system that combines binary observation variables and a hidden Gauss–Markov state process, extending both binary principal component analysis and the classical linear dynamic systems. A BDS learning algorithm, inspired by the popular dynamic texture, and a dissimilarity measure between BDSs, which generalizes the Binet–Cauchy kernel, are introduced. To enable the recognition of highly non-stationary activities, the BDS is embedded in a bag of words. An algorithm is introduced for learning a BDS codebook, enabling the use of the BDS as a visual word for attribute dynamics (WAD). Short-term video segments are then quantized with a WAD codebook, allowing the representation of video as a bag-of-words for attribute dynamics. Video sequences are finally encoded as vectors of locally aggregated descriptors, which summarize the first moments of video snippets on the BDS manifold. Experiments show that this representation achieves state-of-the-art performance on the tasks of complex activity recognition and event identification.

Appendices

Appendices

1.1 Appendix 1: Convergence of Bag-of-Models Clustering

The bag-of-models clustering procedure of Algorithm 2 is a general framework for clustering examples in a Riemannian manifold ${{\mathcal {M}}}$ of statistical models. The goal is to find a preset number of models $\{M_j\}_{j=1}^{K}\subset {{\mathcal {M}}}$ in the manifold that best explain a corpora ${{\mathcal {D}}}=\{\varvec{z}_i\}_{i=1}^N$ ($\varvec{z}_i\in {{\mathcal {Z}}}, \forall i$). It is assumed that all models $M$ are parametrized by a set of parameters ${\varvec{\theta }}$ and have smooth likelihood functions (derivatives of all orders exist and are bounded), and that Algorithm 2 satisfies the following conditions.

Condition 1: the operation $f_{{{\mathcal {M}}}}$ of (20) consists of estimating the parameters ${\varvec{\theta }}$ of ${{\mathcal {M}}}$ by the maximum likelihood estimation (MLE) principle.

Condition 2: the Riemannian metric of the manifold ${{\mathcal {M}}}$ defined by the Fisher information ${{\mathcal {I}}}_{{\varvec{\theta }}_{\varvec{z}}}$ (Jaakkola and Haussler 1999; Amari and Nagaoka 2000) is used as the dissimilarity measure of (21). More precisely, the metric of ${{\mathcal {M}}}$ in the neighbrhood of model $M_{\varvec{z}}$ is

$$\begin{aligned} d_{{{\mathcal {M}}}}(M^*,~M_{\varvec{z}}) = ||{\varvec{\theta }}^* - {\varvec{\theta }}_{\varvec{z}}||^2_{{{\mathcal {I}}}_{{\varvec{\theta }}_{\varvec{z}}}}, \end{aligned}$$

(80)

where $||{\varvec{\theta }}_1-{\varvec{\theta }}_2||^2_{\mathcal {I}}= ({\varvec{\theta }}_1-{\varvec{\theta }}_2)^{\intercal }{\mathcal {I}}({\varvec{\theta }}_1-{\varvec{\theta }}_2)$, and the Fisher information ${{\mathcal {I}}}_{{\varvec{\theta }}_{\varvec{z}}}$ is defined as (Si 1998)

$$\begin{aligned} {{\mathcal {I}}}_{{\varvec{\theta }}_{\varvec{z}}} = -{\mathbb {E}}_{{\varvec{x}}\sim p({\varvec{x}}; {\varvec{\theta }}_{\varvec{z}})}\left[ \nabla ^2_{{\varvec{\theta }}}\ln p({\varvec{x}}; {\varvec{\theta }})|_{{\varvec{\theta }}={\varvec{\theta }}_{\varvec{z}}} \right] . \end{aligned}$$

(81)

Given the similarity between Algorithm 2 and k-means, the convergence of the former can be studied with the techniques commonly used to show that the latter converges. This requires the definition of a suitable objective function to quantify the quality of the fit of the set $\{M_i\}_{j=1}^{K}$ to the corpora ${{\mathcal {D}}}$. We rely on the objective

$$\begin{aligned} \zeta (\{M_i\}_{j=1}^{K}, \{S_j\}_{j=1}^K) = \sum \limits _{j}\sum \limits _{\varvec{z}\in S_j} \ln p_{M_j}(\varvec{z}), \end{aligned}$$

(82)

where $p_{M}(\cdot )$ is the likelihood function of model $M$, and $S_j$ a subset of ${{\mathcal {D}}}$, containing all examples assigned to j-th model. Note that this implies that $\forall i\ne j, S_i\bigcap S_j=\varnothing$ and $\bigcup \limits _jS_j={{\mathcal {D}}}$. From the assumption of smooth models $M$ (i.e., $\forall \varvec{z}\in {{\mathcal {Z}}}, M\in {{\mathcal {M}}}$, $p_M(\varvec{z})<\infty$) and the fact that there is only a finite set of assignments $\{S_j\}_{j=1}^K$, the objective function of (82) is upper bounded. Since the refinement step of Algorithm 2 updates the models so that

$$\begin{aligned} M^{(t+1)}_j = f_{{{\mathcal {M}}}}(S^{(t+1)}_j) = \mathop {\mathrm {argmax}}\limits _{M\in {{\mathcal {M}}}} \sum \limits _{\varvec{z}\in S^{(t+1)}_j}\ln p_M(\varvec{z}), \end{aligned}$$

the objective either increases or remains constant after each refinement step. It remains to prove that the same holds for each assignment step. If that is the case, Algorithm 2 produces a monotonically increasing and upper-bounded sequence of objective function values. By the monotone convergence theorem, this implies that algorithm converges in a finite number of steps. Note that, as in k-means, there is no guarantee on convergence to the global optimum.

It thus remains to prove that the objective of (82) increases with each assignment step. The Riemannian structure of the manifold ${{\mathcal {M}}}$, makes this proof more technical than the corresponding one for k-means. In what follows, we provide a sketch of the proof. Let $M^*$ be the model (of parameters ${\varvec{\theta }}^*$) to which example $\varvec{z}$ is assigned by the assignment step of Algorithm 2, i.e.,

$$\begin{aligned} M^* = \mathop {\mathrm {argmin}}\limits _{M\in \{M^{(t)}_j\}_{j=1}^K}d_{{{\mathcal {M}}}} (M_{\varvec{z}},M) \end{aligned}$$

(83)

and $M^\circ$ (of parameter ${\varvec{\theta }}^\circ$) the equivalent model of the previous iteration. It follows from Condition 2 that

$$\begin{aligned}&d_{{{\mathcal {M}}}}(M^*,~M_{\varvec{z}}) = ||{\varvec{\theta }}^* - {\varvec{\theta }}_{\varvec{z}}||^2_{{{\mathcal {I}}}_{{\varvec{\theta }}_{\varvec{z}}}} \nonumber \\&\quad \leqslant d_{{{\mathcal {M}}}}(M^\circ ,~M_{\varvec{z}}) = ||{\varvec{\theta }}^\circ - {\varvec{\theta }}_{\varvec{z}}||^2_{{{\mathcal {I}}}_{{\varvec{\theta }}_{\varvec{z}}}}. \end{aligned}$$

(84)

Note that, $M_{\varvec{z}}$ is the model $p(\varvec{z}; {\varvec{\theta }}_{\varvec{z}})$ onto which $\varvec{z}$ is mapped by (20). From Condition 1, ${\varvec{\theta }}_{\varvec{z}}=\mathop {\mathrm {argmax}}\nolimits _{{\varvec{\theta }}} p(\varvec{z}; {\varvec{\theta }})$ and, using a Taylor series expansion,

$$\begin{aligned} \ln p(\varvec{z}; {\varvec{\theta }})&\approx \ln p(\varvec{z}; {\varvec{\theta }}_{\varvec{z}}) + \left<\nabla _{\varvec{\theta }}\ln p(\varvec{z}; {\varvec{\theta }})|_{{\varvec{\theta }}={\varvec{\theta }}_{\varvec{z}}}, {\varvec{\theta }}- {\varvec{\theta }}_{\varvec{z}}\right> \nonumber \\& + \frac{1}{2}||{\varvec{\theta }}- {\varvec{\theta }}_{\varvec{z}}||^2_{H_{{\varvec{\theta }}_{\varvec{z}}}} \end{aligned}$$

(85)

$$\begin{aligned}&= \ln p(\varvec{z}; {\varvec{\theta }}_{\varvec{z}}) + \frac{1}{2}||{\varvec{\theta }}- {\varvec{\theta }}_{\varvec{z}}||^2_{H_{{\varvec{\theta }}_{\varvec{z}}}} , \end{aligned}$$

(86)

where $H_{{\varvec{\theta }}_{\varvec{z}}}=\nabla ^2_{\varvec{\theta }}\ln p(\varvec{z}; {\varvec{\theta }})|_{{\varvec{\theta }}={\varvec{\theta }}_{\varvec{z}}}$ is the Hessian of $\ln p(\varvec{z}; {\varvec{\theta }})$ at ${\varvec{\theta }}_{\varvec{z}}$. Since $p(\varvec{z}; {\varvec{\theta }}_{\varvec{z}})$ is the model obtained from a single example $\varvec{z}$, it is a heavily peaky distribution centered at $\varvec{z}$. Hence, the expectation of (81) can be approximated by

$$\begin{aligned} {{\mathcal {I}}}_{{\varvec{\theta }}_{\varvec{z}}} \approx -H_{{\varvec{\theta }}_{\varvec{z}}}. \end{aligned}$$

(87)

Combining (84), (86), and (87) then results in

$$\begin{aligned} \ln p(\varvec{z}; {\varvec{\theta }}^*)&\approx \ln p(\varvec{z}; {\varvec{\theta }}_{\varvec{z}}) + \frac{1}{2}||{\varvec{\theta }}^* - {\varvec{\theta }}_{\varvec{z}}||^2_{H_{{\varvec{\theta }}_{\varvec{z}}}} \\&\approx \ln p(\varvec{z}; {\varvec{\theta }}_{\varvec{z}}) - \frac{1}{2}||{\varvec{\theta }}^* - {\varvec{\theta }}_{\varvec{z}}||^2_{{{\mathcal {I}}}_{{\varvec{\theta }}_{\varvec{z}}}}\\&\geqslant \ln p(\varvec{z}; {\varvec{\theta }}_{\varvec{z}}) - \frac{1}{2}||{\varvec{\theta }}^\circ - {\varvec{\theta }}_{\varvec{z}}||^2_{{{\mathcal {I}}}_{{\varvec{\theta }}_{\varvec{z}}}} \approx \ln p(\varvec{z}; {\varvec{\theta }}^\circ ). \end{aligned}$$

It follows that the objective of (82) increases after each assignment step. This is intuitive in the sense that, the closer a model $M$ is to an example’s representative model, the better $M$ can explain that example.

Appendix 2: Optimization

In this appendix, we derive (72), by considering the optimization problem

$$\begin{aligned} X^* = \mathop {\mathrm {argmax}}\limits _{X\in {{\mathcal {S}}}_{++}}&~b\ln \left| X\right| -\mathrm {tr}(AX), \\ s.t.&~A\in {{\mathcal {S}}}_{++},~b>0. \nonumber \end{aligned}$$

(88)

Since 1) both $b\ln \left| X\right|$ and $-\mathrm {tr}(AX)$ are smooth and concave functions in X (Boyd and Vandenberghe 2004), and 2) the domain ${{\mathcal {S}}}_{++}$ is an open convex set, the supremum of (88) is achieved at either 1) its stationary point(s) (if any), or 2) the boundary of its domain. The derivative of the objective function of (88) is

$$\begin{aligned} \frac{\partial }{\partial X}&\big \{b\ln \left| X\right| - \mathrm {tr}(AX)\big \} = b({X}^{-1})^{\intercal } - A. \end{aligned}$$

(89)

Setting (89) to zero leads to

$$\begin{aligned} {X}^{*} = bA^{-1} \in {{\mathcal {S}}}_{++}. \end{aligned}$$

(90)

Applying this result to (71), with $b=1$, $X={\varSigma }$, and $A=W$, leads to (72).

Appendix 3: Variational Inference for BDS

The key computation of the variational inference procedure of Sect. 6.2.3 is to determine

$$\begin{aligned}&\varvec{m}_t =\left\langle \varvec{x}_t\right\rangle _{q}, \\&{\varSigma }_{t,t} = \left\langle (\varvec{x}_t-\varvec{m}_t)(\varvec{x}_t-\varvec{m}_t)^{\intercal }\right\rangle _{q}, \\&{\varSigma }_{t,t+1} = \left\langle (\varvec{x}_t-\varvec{m}_t)(\varvec{x}_{t+1}-\varvec{m}_{t+1})^{\intercal }\right\rangle _{q}. \end{aligned}$$

In this appendix, we derive an efficient method for this computation, which draws on the solution of the identical variational inference problem for the LDS of (10). We start by discussing the LDS case.

1.1 (a) Inference for Linear Dynamic Systems

Consider the LDS of (10) with parameters ${\varvec{\theta }}_{LDS}=\{S, {\varvec{\mu }}, A, C, Q, {R} , \varvec{u}\}$, an observation sequence $\{{\varvec{y}}_1^\tau \}$ (${\varvec{y}}_t\in {\mathbb {R}}^K$), and the variational distribution $q(\varvec{x})$ of (57). Similarly to the derivation of Sect. 6.2.3, the variational lower bound of (41) for the log-likelihood of the LDS can be shown to be

$$\begin{aligned} {\mathscr {L}}({\varvec{\theta }}, {\varvec{y}}, q)= & {} \left\langle \ln p(\varvec{x}_1)\right\rangle _{q} + \sum _{t=1}^{\tau -1} \left\langle \ln p(\varvec{x}_{t+1}|\varvec{x}_t)\right\rangle _{q} \nonumber \\&+ \sum _{t=1}^{\tau } \left\langle \ln p({\varvec{y}}_t|\varvec{x}_t)\right\rangle _{q} + H_{q}(X), \end{aligned}$$

(91)

with $\left\langle \ln p(\varvec{x}_1)\right\rangle _{q}, \left\langle \ln p(\varvec{x}_{t+1}|\varvec{x}_t)\right\rangle _{q},$ and $H_{q}(X)$ as in (63)-(65). Furthermore, defining ${{\tilde{{\varvec{y}}}}}_t ={\varvec{y}}_t - \varvec{u}$,

$$\begin{aligned} \left\langle \ln p({\varvec{y}}_t|\varvec{x}_t)\right\rangle _{q}= & {} \left\langle \ln {\mathcal {G}}({{\tilde{{\varvec{y}}}}}_t;C\varvec{x}_t,R)\right\rangle _{q(\varvec{x}_t)} \nonumber \\= & {} \left\langle \ln {\mathcal {G}}(C\varvec{x}_t;{{\tilde{{\varvec{y}}}}}_t,R)\right\rangle _{{\mathcal {G}}(\varvec{x}_t; \varvec{m}_t, {\varSigma }_{t,t})} \nonumber \\= & {} \left\langle \ln {\mathcal {G}}(\varvec{x}_t;{{\tilde{{\varvec{y}}}}}_t,R)\right\rangle _{{\mathcal {G}}(\varvec{x}_t; C\varvec{m}_t, C{\varSigma }_{t,t}C^{\intercal })} \end{aligned}$$

(92)

and, from (8),

$$\begin{aligned} \left\langle \ln p({\varvec{y}}_t|\varvec{x}_t)\right\rangle _{q}\propto & {} -\frac{1}{2}\Big [||{{\tilde{{\varvec{y}}}}}_t-C\varvec{m}_t||^2_{ {R} } + \mathrm {tr}( {R} ^{-1}C{\varSigma }_{t,t}C^{\intercal }) \Big ]. \end{aligned}$$

It follows that

$$\begin{aligned} {\mathscr {L}}({\varvec{\theta }}, {\varvec{y}}, q)\propto & {} -\frac{1}{2}\bigg \{ ||{\varvec{\mu }}-\varvec{m}_1||^2_{S} +\mathrm {tr}(S^{-1}{\varSigma }_{1,1}) \nonumber \\+ & {} \sum _{t=1}^{\tau -1} \mathrm {tr}({\varGamma }^{-1}{\varPhi }_t) + \sum _{t=1}^{\tau } \mathrm {tr}( {R} ^{-1}C{\varSigma }_{t,t}C^{\intercal }) \nonumber \\+ & {} \sum _{t=1}^{\tau } ||{{\tilde{{\varvec{y}}}}}_t-C\varvec{m}_t||^2_{ {R} } \bigg \} + \frac{1}{2}\ln \left| {\varSigma }\right| , \end{aligned}$$

(93)

where ${\varGamma }$ and ${\varPhi }_t$ are defined in Sect. 6.2.3.

As was the the case with (69), the optimization of (93) with respect to the variational distribution $q$ can be factorized into two optimization problems

$$\begin{aligned} \{\varvec{m}^*, {\varSigma }^*\} =&\mathop {\mathrm {argmax}}\limits _{\{\varvec{m}, {\varSigma }\}\in {\mathbb {R}}^{L\tau }\times {{\mathcal {S}}}^{L\tau }_{++}} {\mathscr {L}}({\varvec{\theta }}, {\varvec{y}}, q) \\ =&~\bigg \{ \mathop {\mathrm {argmax}}\limits _{\varvec{m}\in {\mathbb {R}}^{L\tau }} {\mathscr {L}}({\varvec{\theta }}, {\varvec{y}}, q), \mathop {\mathrm {argmax}}\limits _{{\varSigma }\in {{\mathcal {S}}}^{L\tau }_{++}} {\mathscr {L}}({\varvec{\theta }}, {\varvec{y}}, q) \bigg \}. \end{aligned}$$

In fact, the dependence of (93) on ${\varSigma }$ is identical to that of (69), up to the replacement of R by 4I. Hence, the optimal ${\varSigma }$ is still the solution of (71), i.e.,

$$\begin{aligned} {\varSigma }^*&= W^{-1}, \end{aligned}$$

(94)

but with a matrix $W\in {{\mathcal {S}}}_{++}$ which is slightly different from (71), namely

$$\begin{aligned} W_{i,j} = {\left\{ \begin{array}{ll} A^{\intercal }Q^{-1}A+ S^{-1}+C^{\intercal } {R} ^{-1}C, &{} i=j=1, \\ A^{\intercal }Q^{-1}A+ Q^{-1}+C^{\intercal } {R} ^{-1}C,&{}1<i=j<\tau , \\ Q^{-1}+C^{\intercal } {R} ^{-1}C, &{} i=j=\tau , \\ -Q^{-1}A, &{} i=j+1, \\ -A^{\intercal }Q^{-1}, &{} i=j-1, \\ {0}, &{} \text {otherwise}, \end{array}\right. } \end{aligned}$$

(95)

where $W_{i,j}\in {\mathbb {R}}^{L\times L}$ is the block in row-i, column-j of $W$.

In summary, the algorithm for learning the covariance of the variational distribution of the BDS is identical to the learning algorithm for the covariance of the variational distribution of a LDS with $R = 4I$. Furthermore, since all random variables $\varvec{x}$ and ${\varvec{y}}$ (as well as all marginal or conditional distributions) of the LDS are Gaussian, the variational inference is exact in this case and

$$\begin{aligned} q^*(\varvec{x})=p(\varvec{x}|{\varvec{y}}; {\varvec{\theta }}_{LDS}). \end{aligned}$$

It thus follows that the standard algorithms for exact inference of $p(\varvec{x}|{\varvec{y}}; {\varvec{\theta }}_{LDS})$ with the LDS can be used to compute the covariance ${\varSigma }^*$ of the variational distribution of the BDS. In the following section, we briefly review the Kalman smoothing filter, which is the most popular such algorithm.

The situation is, however, different for the mean of the variational distribution. In this case, the LDS continues to have a simple closed-form solution, namely

$$\begin{aligned} \varvec{m}^*&= W^{-1}\varvec{\nu }, \end{aligned}$$

(96)

where $W$ is as in (95) and

$$\begin{aligned} \varvec{\nu }= \begin{bmatrix} \varvec{\nu }_1\\ \vdots \\ \varvec{\nu }_\tau \end{bmatrix},~~ \varvec{\nu }_t = {\left\{ \begin{array}{ll} S^{-1}{\varvec{\mu }}+ C {R} ^{-1}{{\tilde{{\varvec{y}}}}}_1, &{} t=1, \\ C {R} ^{-1}{{\tilde{{\varvec{y}}}}}_t, &{} 1<t\leqslant \tau . \end{array}\right. } \end{aligned}$$

(97)

However, because the dependence of (93) on $\varvec{m}$ is no longer identical to that of (69), the LDS solution is not informative for learning the BDS. A different procedure is thus required to learn the variational mean of the BDS. This is discussed in Sect. 1.

1.2 (b) Inferring the Variational Covariance ${\varSigma }$ of the BDS

Note that, $\varvec{m}^*$ and ${\varSigma }^*$ have size linear and quadratic, respectively, in $\tau$, the length of the sequence $\{{\varvec{y}}_1^\tau \}$. This makes the direct solution of (96) and (94) expensive for long sequences - complexity $O(L^\rho \tau ^\rho )$ with $\rho \approx 2.4$. Furthermore, this solution is unnecessary, since inference with both the LDS and BDS only requires ${\varSigma }_{t,t}^*$ and ${\varSigma }_{t,t+1}^*$. A popular efficient alternative is the Kalman smoothing filter (Shumway and Stoffer 1982; Roweis and Ghahramani 1999), which is commonly use to estimate the posteriors $p(\varvec{x}_t|{\varvec{y}}_{1}^{\tau })$ and $p(\varvec{x}_t, \varvec{x}_{t+1}|{\varvec{y}}_{1}^{\tau })$ of the LDS, i.e., $\varvec{m}^*$ of (96), ${\varSigma }^*_{t,t}$ and ${\varSigma }^*_{t,t+1}$ of (94).

Defining expectations conditioned on the observed sequence from time $t=1$ to $t=r$ as

$$\begin{aligned} \hat{\varvec{x}}_t^{r}= & {} \left\langle \varvec{x}_t\right\rangle _{p(\varvec{x}_t|{\varvec{y}}_1,\ldots ,{\varvec{y}}_r)}, \end{aligned}$$

(98)

$$\begin{aligned} \hat{V}_{t,k}^r= & {} \left\langle (\varvec{x}_t-\hat{\varvec{x}}_t^r)(\varvec{x}_{k}-\hat{\varvec{x}}_{k}^r)^{\intercal }\right\rangle _{p({\varvec{x}_t,\varvec{x}_k|{\varvec{y}}_1,\ldots ,{\varvec{y}}_r})}, \end{aligned}$$

(99)

the estimates are calculated via the forward and backward recursions:

• In the forward recursion, for $t=1, \ldots , \tau$, compute

  $$\begin{aligned} \hat{V}_{t,t}^{t-1}= & {} {A} \hat{V}_{t-1,t-1}^{t-1} {A} ^{\intercal }+ {Q} , \end{aligned}$$

  (100)
  $$\begin{aligned} K_t= & {} \hat{V}_t^{t-1} {C} ^{\intercal }( {C} \hat{V}_{t,t}^{t-1} {C} ^{\intercal }+ {R} _t)^{-1}, \end{aligned}$$

  (101)
  $$\begin{aligned} \hat{V}_{t,t}^t= & {} \hat{V}_{t,t}^{t-1} - K_t {C} \hat{V}_{t,t}^{t-1}, \end{aligned}$$

  (102)
  $$\begin{aligned} \hat{\varvec{x}}_t^{t-1}= & {} {A} \hat{\varvec{x}}_{t-1}^{t-1}, \end{aligned}$$

  (103)
  $$\begin{aligned} \hat{\varvec{x}}_t^t= & {} \hat{\varvec{x}}_t^{t-1} + K_t ({{\tilde{{\varvec{y}}}}}_t- {C} \hat{\varvec{x}}_t^{t-1}), \end{aligned}$$

  (104)

  with initial conditions $\hat{\varvec{x}}_1^0 = {\varvec{\mu }}$ and $\hat{V}_{1,1}^0= {S}$.

• In the backward recursion, for $t=\tau , \ldots ,1$,

  $$\begin{aligned} J_{t-1}= & {} \hat{V}_{t-1,t-1}^{t-1} {A} ^{\intercal }(\hat{V}_{t,t}^{t-1})^{-1}, \end{aligned}$$

  (105)
  $$\begin{aligned} \hat{\varvec{x}}_{t-1}^\tau= & {} \hat{\varvec{x}}_{t-1}^{t-1}+J_{t-1}(\hat{\varvec{x}}_t^\tau - {A} \hat{\varvec{x}}_{t-1}^{t-1}), \end{aligned}$$

  (106)
  $$\begin{aligned} \hat{V}_{t-1,t-1}^\tau= & {} \hat{V}_{t-1,t-1}^{t-1}+J_{t-1}(\hat{V}_{t,t}^\tau -\hat{V}_{t,t}^{t-1})J_{t-1}^{\intercal }, \end{aligned}$$

  (107)

  and for $t=\tau ,\ldots ,2$,

  $$\begin{aligned}&\hat{V}_{t-1,t-2}^\tau = ~\hat{V}_{t-1,t-1}^{t-1}J_{t-2}^{\intercal } \nonumber \\&\quad ~ + J_{t-1}(\hat{V}_{t,t-1}^\tau - {A} \hat{V}_{t-1,t-1}^{t-1})J_{t-2}^{\intercal } \end{aligned}$$

  (108)

  with initial c ondition $\hat{V}_{\tau ,\tau -1}^\tau = (I-K_\tau {C} ) {A} \hat{V}_{\tau -1,\tau -1}^{\tau -1}$.

This algorithm can be used to efficiently compute the variational covariance parameters ${\varSigma }_{t,t}^*$ and ${\varSigma }_{t,t+1}^*$ of the BDS, which are exactly the matrices $\hat{V}_{t,t}^\tau$ of (107) and $\hat{V}_{t,t-1}^\tau$ of (108), respectively. This has complexity $O(L^\rho \tau ),$ with $\rho \approx 2.4$.

1.3 (c) Infering the Variational mean $\varvec{m}$ of the BDS

The variational mean $\varvec{m}$ is the solution of

$$\begin{aligned} \varvec{m}^* =&~\mathop {\mathrm {argmax}}_{\varvec{m}}~{{\hat{{\mathscr {L}}}}}({\varvec{\theta }}, q) \nonumber \\ =&~\mathop {\mathrm {argmax}}_{\varvec{m}} \bigg \{ {\varvec{\mu }}_0^{\intercal }S^{-1}\varvec{m}_1 -\frac{1}{2} \varvec{m}_1^{\intercal }S^{-1}\varvec{m}_1\nonumber \\&~~-\frac{1}{2} \sum _{t=1}^{\tau -1} \lambda _t^{\intercal }{\varGamma }^{-1}\lambda _t \nonumber \\&~~+ \sum _{t,k}\Big [ {\pi }_{kt}\ln \sigma ({\hat{{\omega }}}_{kt}) + (1-{\pi }_{kt})\ln \sigma (-{\hat{{\omega }}}_{kt})\Big ]\bigg \}. \end{aligned}$$

(109)

This can be rewritten as

$$\begin{aligned} \varvec{m}^* =&~\mathop {\mathrm {argmax}}_{\varvec{m}} \bigg \{-\frac{1}{2}\varvec{m}^{\intercal }{{\tilde{W}}}\varvec{m}+ \varvec{ b}_1^{\intercal }\varvec{m}_1 \nonumber \\&+ \sum _{t, k} \Big [{\pi }_{kt}\ln \sigma ({\hat{{\omega }}}_{kt}) + (1-{\pi }_{kt})\ln \sigma (-{\hat{{\omega }}}_{kt}) \Big ] \bigg \}, \end{aligned}$$

(110)

where

$$\begin{aligned} {{\tilde{W}}}_{i,j} = {\left\{ \begin{array}{ll} A^{\intercal }Q^{-1}A+ S^{-1}, &{} i=j=1, \\ A^{\intercal }Q^{-1}A+ Q^{-1},&{}1<i=j<\tau , \\ Q^{-1}, &{} i=j=\tau , \\ -Q^{-1}A, &{} i=j+1, \\ -A^{\intercal }Q^{-1}, &{} i=j-1, \\ {0}, &{} \text {otherwise}, \end{array}\right. } \end{aligned}$$

(111)

${\hat{{\omega }}}_{kt}=C_{k\cdot }\varvec{m}_t+u_k$, and $\varvec{ b}_1 = 2S^{-1}{\varvec{\mu }}$. Since ${{\hat{{\mathscr {L}}}}}({\varvec{\theta }}, q)$ is a concave function of $\varvec{m}\in {\mathbb {R}}^{\tau L}$, gradient-based methods can be applied to search for the stationary point where global optimum is guaranteed.

The gradient of ${{\hat{{\mathscr {L}}}}}({\varvec{\theta }}, q)$ is

$$\begin{aligned} \frac{\partial }{\partial \varvec{m}}{{\hat{{\mathscr {L}}}}}({\varvec{\theta }}, q) =&~ -{{\tilde{W}}}\varvec{m}+ \begin{bmatrix} \varvec{ b}_1\\ {\varvec{0}} \end{bmatrix} - \begin{bmatrix} C^{\intercal }&\\&\ddots&\\&C^{\intercal } \end{bmatrix} \begin{bmatrix} {\varvec{\beta }}_1 \\ \vdots \\ {\varvec{\beta }}_\tau \end{bmatrix}, \end{aligned}$$

(112)

where

$$\begin{aligned} {\varvec{\beta }}_t =&~ \begin{bmatrix} \sigma ({\hat{{\omega }}}_{kt})-{\pi }_{1t}\\ \vdots \\ \sigma ({\hat{{\omega }}}_{Kt})-{\pi }_{Kt} \end{bmatrix}. \end{aligned}$$

The second-order partial derivatives of ${{\hat{{\mathscr {L}}}}}({\varvec{\theta }}, q)$ is

$$\begin{aligned} \frac{\partial ^2}{\partial \varvec{m}^2}{{\hat{{\mathscr {L}}}}}({\varvec{\theta }}, q)&= -{{\tilde{W}}} - \begin{bmatrix} C^{\intercal }{\varXi }_1C&\\&\ddots&\\&C^{\intercal }{\varXi }_\tau C\end{bmatrix}, \end{aligned}$$

(113)

where

$$\begin{aligned} {\varXi }_t = {\mathrm {diag}}(\sigma ({\hat{{\omega }}}_{1t})\sigma (-{\hat{{\omega }}}_{1t}),~\ldots ,~\sigma ({\hat{{\omega }}}_{Kt})\sigma (-{\hat{{\omega }}}_{Kt})). \end{aligned}$$

Given the concavity and smoothness of ${{\hat{{\mathscr {L}}}}}({\varvec{\theta }}, q)$, many popular numerical optimization algorithms can be utilized to search for its optimum, e.g., gradient descent, Newton–Raphson method, Broyden–Fletcher–Goldfarb–Shanno (BFGS) algorithm, etc.

Appendix 4: The Fisher Vector for BDS

In this section, we present the derivation of the Fisher vector for BDS using the tightest variational lower bound ${{\hat{{\mathscr {L}}}}}({\varvec{\theta }}, q^*)$ of (69). This consists of computing partial derivatives of ${{\hat{{\mathscr {L}}}}}({\varvec{\theta }}, q^*)$ w.r.t. each of the BDS parameters ${\varvec{\theta }}=\{S^{-1}, {\varvec{\mu }}, A, Q^{-1}, C, \varvec{u}\}$.

1.1 (a) Derivative w.r.t. $S^{-1}$

We have

$$\begin{aligned}&\frac{\partial }{\partial S^{-1}} {{\hat{{\mathscr {L}}}}}({\varvec{\theta }}, q^*) \nonumber \\&\quad = \frac{\partial }{\partial S^{-1}}~\frac{1}{2}\bigg [\ln \left| S^{-1}\right| -\mathrm {tr}\Big (({\hat{P}}_{1,1}^*- 2\varvec{m}^*_1{\varvec{\mu }}^{\intercal }+ {\varvec{\mu }}{\varvec{\mu }}^{\intercal })S^{-1}\Big ) \bigg ] \nonumber \\&\quad =\frac{1}{2}\Big (S+ 2{\varvec{\mu }}\varvec{m}_1^{*T} - {\hat{P}}_{1,1}^* - {\varvec{\mu }}{\varvec{\mu }}^{\intercal } \Big ), \end{aligned}$$

(114)

where ${\hat{P}}_{t_1,t_2}^*$ is defined in (62). Note that, $S^{-1}\in {{\mathcal {S}}}^L_{++}$, thus the derivative of (114) needs to be projected into the space of symmetric matrices ${{\mathcal {S}}}^L$. Since an orthonormal basis of ${{\mathcal {S}}}^L$ is $\{\frac{1}{2}(E_{i,j}+E_{j,i}), 1\leqslant i\leqslant j\leqslant L\}$, where $E_{i,j}\in {\mathbb {R}}^{L\times L}$ with the (i, j)-element equal to one and all the rest elements being zero, it can be shown that after the projection, (114) becomes

$$\begin{aligned} \frac{\partial }{\partial S^{-1}}&{{\hat{{\mathscr {L}}}}}({\varvec{\theta }}, q^*) \nonumber \\ =&~\frac{1}{2}\Big (S+ {\varvec{\mu }}\varvec{m}_1^{*\intercal } + \varvec{m}^*_1{\varvec{\mu }}^{\intercal } - {\hat{P}}_{1,1}^* - {\varvec{\mu }}{\varvec{\mu }}^{\intercal } \Big ). \end{aligned}$$

(115)

1.2 (b) Derivative w.r.t. ${\varvec{\mu }}$

We have

$$\begin{aligned} \frac{\partial }{\partial {\varvec{\mu }}}&{{\hat{{\mathscr {L}}}}}({\varvec{\theta }}, q^*) \nonumber \\ =&~\frac{\partial }{\partial {\varvec{\mu }}} \Big [ {\varvec{\mu }}^{\intercal }S^{-1}\varvec{m}^*_1 - \frac{1}{2}{\varvec{\mu }}^{\intercal }S^{-1}{\varvec{\mu }}\Big ] \nonumber \\ =&~S^{-1}(\varvec{m}^*_1 - {\varvec{\mu }}). \end{aligned}$$

(116)

1.3 (c) Derivative w.r.t. $A$

We have

$$\begin{aligned} \frac{\partial }{\partial A}&{{\hat{{\mathscr {L}}}}}({\varvec{\theta }}, q^*) \nonumber \\ =&~\frac{\partial }{\partial A} \bigg [ \sum \limits _{t=1}^{\tau -1} \mathrm {tr}\bigg ( {\hat{P}}_{t,t+1}^*Q^{-1}A-\frac{1}{2}{\hat{P}}_{t,t}^*A^{\intercal }Q^{-1}A\bigg ) \bigg ] \nonumber \\ =&~\frac{\partial }{\partial A} \bigg [ \mathrm {tr}\bigg ( {\varPsi }^{\intercal }Q^{-1}A-\frac{1}{2}\phi A^{\intercal }Q^{-1}A\bigg ) \bigg ] \nonumber \\ =&~({\varPsi }^{\intercal }Q^{-1})^{\intercal } - \frac{1}{2}\Big [Q^{-\intercal }A\phi ^{\intercal } + Q^{-1}A\phi \Big ] \nonumber \\ =&~Q^{-1}({\varPsi }-A\phi ), \end{aligned}$$

(117)

where

$$\begin{aligned} \phi = \sum \limits _{t=2}^{\tau } {\hat{P}}_{t-1,t-1}^*,~ {\varPsi }= \sum \limits _{t=2}^{\tau } {\hat{P}}_{t,t-1}^*. \end{aligned}$$

1.4 (d) Derivative w.r.t. $Q^{-1}$

We have

$$\begin{aligned} \frac{\partial }{\partial Q^{-1}}&{{\hat{{\mathscr {L}}}}}({\varvec{\theta }}, q^*) \nonumber \\ =&~\frac{\partial }{\partial Q^{-1}} \bigg [ \sum \limits _{t=1}^{\tau -1} \mathrm {tr}\bigg ( A{\hat{P}}_{t,t+1}^*Q^{-1} -\frac{1}{2}A{\hat{P}}_{t,t}^*A^{\intercal }Q^{-1} \nonumber \\& -\frac{1}{2}{\hat{P}}_{t+1,t+1}^*Q^{-1} \bigg ) + (\frac{\tau -1}{2})\ln \left| Q^{-1}\right| \bigg ] \nonumber \\ =&~\frac{\partial }{\partial Q^{-1}} \bigg [ \mathrm {tr}\bigg ( A{\varPsi }^{\intercal }Q^{-1} -\frac{1}{2}A\phi A^{\intercal }Q^{-1} -\frac{1}{2}\varphi Q^{-1} \bigg ) \nonumber \\& + (\frac{\tau -1}{2})\ln \left| Q^{-1}\right| \bigg ] \nonumber \\ =&~{\varPsi }A^{\intercal } + \frac{1}{2}\Big [(\tau -1)Q-A\phi A^{\intercal } - \varphi \Big ], \end{aligned}$$

(118)

where

$$\begin{aligned} \varphi&= \sum \limits _{t=2}^{\tau } {\hat{P}}_{t,t}^*.~ \end{aligned}$$

(119)

Again, since $Q^{-1}\in {{\mathcal {S}}}_{++}$, the partial derivative of (118) is projected into ${{\mathcal {S}}}$, giving

$$\begin{aligned} \frac{\partial }{\partial Q^{-1}}&{{\hat{{\mathscr {L}}}}}({\varvec{\theta }}, q^*) \nonumber \\ =~\frac{1}{2}&\Big [{\varPsi }A^{\intercal }+A{\varPsi }^{\intercal }-A\phi A^{\intercal } - \varphi + (\tau -1)Q\Big ]. \end{aligned}$$

(120)

1.5 D.5 Derivative w.r.t. ${\tilde{C}}$

Assuming ${\tilde{C}}=\begin{bmatrix} C&\varvec{u}\end{bmatrix}$, we have

$$\begin{aligned} \frac{\partial }{\partial {\tilde{C}}}&{{\hat{{\mathscr {L}}}}}({\varvec{\theta }}, q^*) \nonumber \\ =&~\frac{\partial }{\partial {\tilde{C}}} \bigg \{ \sum \limits _{k,t} \Big [ {\pi }_{kt}\ln \sigma ({\tilde{C}}_{k\cdot }{\varvec{b}_{t}}) + (1-{\pi }_{kt})\ln \sigma (-{\tilde{C}}_{k\cdot }{\varvec{b}_{t}}) \Big ] \nonumber \\&- \frac{1}{8}\mathrm {tr}\big ({\tilde{C}}{{\tilde{{\varUpsilon }}}} {\tilde{C}}^{\intercal }\big ) \bigg \} \nonumber \\ =&~-\frac{1}{4} \bigg \{{\tilde{C}}{{\tilde{{\varUpsilon }}}} + \sum _{t=1}^{\tau } \begin{bmatrix} \sigma ({\tilde{C}}_{1\cdot }{\varvec{b}_{t}}) - {\pi }_{1t}\\ \vdots \\ \sigma ({\tilde{C}}_{K\cdot }{\varvec{b}_{t}}) - {\pi }_{Kt} \end{bmatrix} {\varvec{b}}_{t}^{\intercal }\Bigg \}, \end{aligned}$$

(121)

where ${\tilde{C}}_{k\cdot }$ is the k-th row of ${\tilde{C}}$, and

$$\begin{aligned} {{\tilde{{\varUpsilon }}}} = \begin{pmatrix} \sum \limits _{t=1}^{\tau } \varSigma _{t,t}^{*} &{} 0\\ 0 &{} 0 \end{pmatrix},~ {\varvec{b}}_{t} = \begin{pmatrix} \varvec{m}^*_t\\ 1 \end{pmatrix}. \end{aligned}$$

Appendix 5: Weizmann Complex Activity

1.1 (a) Synthetic Datasets

The synthetic dataset contains three sets: Syn-4/5/6, Syn20 $\times$ 1 and Syn1s0 $\times$ 2, which are generated using the 10 atomic actions (per person) from the original Weizmann dataset by Gorelick et al. (2007). Exemplar activities in Syn-4/5/6, Syn20 $\times$ 1, and Syn10 $\times$ 2 are shown in Tables 7, 8, and 9, respectively. For Syn20 $\times$ 1, and Syn10 $\times$ 2, two of the 9 instances for an activity (each instance is assembled from each of the 9 people’s atomic actions).

Table 7 Examples for Syn-4/5/6

Table 8 Examples for Syn20 $\times$ 1

Table 9 Examples for Syn10 $\times$ 2

Table 10 Attributes for Weizmann actions

Appendix 6: Attribute Definition

1.1 (a) Weizmann Complex Activity

Attribute definitions from (Liu et al. 2011) on Weizmann complex activity are shown in Table 10.

1.2 (b) Olympic Sports

Attribute definitions from (Liu et al. 2011) on Olympic Sports dataset (Niebles et al. 2010) are shown in Table 11.

Table 11 Attributes for Olympic Sports

Table 12 Attribute list for TRECVID MED11

1.3 (c) TRECVID MED11

Attribute definitions from (Bhattacharya 2013) on TRECVID MED11 dataset Over et al. (2011) are shown in Table 12.