Cross the data desert: generating textual-visual summary on the evolutionary microblog stream

Abstract

Effectively and efficiently summarizing social media is crucial and non-trivial to analyze social media. On social streams, events which are the main concept of semantic similar social messages, often bring us a firsthand story of daily news. However, to identify the valuable news, it is almost impossible to plough through millions of multi-modal messages one by one with traditional methods. Thus, it is urgent to summarize events with a few representative data samples on the streams. In this paper, we provide a vivid textual-visual media summarization approach for microblog streams, which exploits the incremental latent semantic analysis (LSA) of detected events. Firstly, with a novel weighting scheme for keyword relationship, we can detect and track daily sub-events on a keyword relation graph (WordGraph) of microblog streams effectively. Then, to summarize the stream with representative texts and images, we use cross-modal fusion to analyze the semantics of microblog texts and images incrementally and separately, with a novel incremental cross-modal LSA algorithm. The experimental results on a real microblog dataset show that our method is at least 1.31% better and 23.67% faster than existing state-of-the-art methods, and cross-modal fusion can improve the summarization performance by 4.16% on average.

Appendix: The derivation of incremental LSA

In Section 3.2.2, the incremental summarization of evolving events is based on the incremental LSA of $\mathbf {B}^{t + 1}$, which means it should be specified by the LSA of $\mathbf {B}^{t}$ (U^tΣ^tV^tT) and the feature matrix $\mathbf {{\Delta }}$ of new data in day $t + 1$. The solution of (11) is based on the algorithm in [9]. It uses the quasi-Gram-Schmidt algorithm [34] to perform a quick SVD on a large matrix. The entire process consists of two parts: the reconstruction of $\mathbf {B}^{t + 1}$ based on QR factorization, and the fast SVD of $\mathbf {B}^{t + 1}$ based on the quasi-Gram-Schmidt.

First, the $\mathbf {B}^{t + 1}$ of (10) should be rewritten in the production format, since its SVD cannot be represented by the sum of the SVDs of several sub-matrices. By substituting $\mathbf {B}^{t}$ with its LSA, $\mathbf {B}^{t + 1}$ can be decomposed as:

$$ \begin{array}{lllll} \mathbf{B}^{t + 1} & = \mathbf{U}^{t} \mathbf{{\Sigma}}^{t} {\mathbf{V}^{t}}^{T} + \mathbf{I} \cdot \mathbf{{\Delta}} \\ & = [ \mathbf{U}^{t} \quad \mathbf{I} ] \cdot \left[ \begin{array}{llllllll} \mathbf{{\Sigma}}^{t} & \mathbf{0} \\ \mathbf{0} & \mathbf{I} \end{array} \right] \cdot [ \mathbf{V}^{t} \quad \mathbf{{\Delta}}^{T} ]^{T} \end{array}. $$

(19)

To decompose $\mathbf {B}^{t + 1}$ further, the QR factorization is performed on the $[\mathbf {U}^{t} \quad \mathbf {I}]$ and $[\mathbf {V}^{t} \quad \mathbf {{\Delta }}^{T}]$ of (19). Supposing $\mathbf {Q}_{\mathrm {I}} \mathbf {R}_{\mathrm {I}} = (\mathbf {I} - \mathbf {U}^{t} {\mathbf {U}^{t}}^{T}) \mathbf {I}$ and $\mathbf {Q}_{\mathrm {{\Delta }}} \mathbf {R}_{\mathrm {{\Delta }}} = (\mathbf {I} - \mathbf {V}^{t} {\mathbf {V}^{t}}^{T}) \mathbf {{\Delta }}^{T}$ are two QR factorizations, the matrix $\mathbf {I}$ can be specified by $\mathbf {U}^{t} {\mathbf {U}^{t}}^{T} \mathbf {I} + \mathbf {Q}_{\mathrm {I}} \mathbf {R}_{\mathrm {I}}$ and $\mathbf {{\Delta }}^{T}$ by $\mathbf {V}^{t} {\mathbf {V}^{t}}^{T} \mathbf {{\Delta }}^{T} + \mathbf {Q}_{\mathrm {{\Delta }}} \mathbf {R}_{\mathrm {{\Delta }}}$. Consequently, the $\mathbf {B}^{t + 1}$ of (19) is rewritten as:

$$ \begin{array}{lllll} \mathbf{B}^{t + 1} & = [ \mathbf{U}^{t} \quad \mathbf{U}^{t} {\mathbf{U}^{t}}^{T} \mathbf{I} + \mathbf{Q}_{\mathrm{I}} \mathbf{R}_{\mathrm{I}} ] \left[ \begin{array}{llllllll} \mathbf{{\Sigma}}^{t} & \mathbf{0} \\ \mathbf{0} & \mathbf{I} \end{array} \right] [ \mathbf{V}^{t} \quad \mathbf{V}^{t} {\mathbf{V}^{t}}^{T} \mathbf{{\Delta}}^{T} + \mathbf{Q}_{\mathrm{{\Delta}}} \mathbf{R}_{\mathrm{{\Delta}}} ]^{T} \\ & = [ \mathbf{U}^{t} \quad \mathbf{Q}_{\mathrm{I}} ] \left[ \begin{array}{llllllll} \mathbf{I} & {\mathbf{U}^{t}}^{T} \mathbf{I} \\ \mathbf{0} & \mathbf{R}_{\mathrm{I}} \end{array} \right] \left[ \begin{array}{llllllll} \mathbf{{\Sigma}}^{t} & \mathbf{0} \\ \mathbf{0} & \mathbf{I} \end{array} \right] \left[ \begin{array}{llllllll} \mathbf{I} & {\mathbf{V}^{t}}^{T} \mathbf{{\Delta}}^{T} \\ \mathbf{0} & \mathbf{R}_{\mathrm{{\Delta}}} \end{array} \right]^{T} [ \mathbf{V}^{t} \quad \mathbf{Q}_{\mathrm{{\Delta}}} ]^{T} \\ & = [ \mathbf{U}^{t} \quad \mathbf{Q}_{\mathrm{I}} ] \left( \left[ \begin{array}{llllllll} \mathbf{{\Sigma}}^{t} & \mathbf{0} \\ \mathbf{0} & \mathbf{0} \end{array} \right] + \left[ \begin{array}{llllllll} {\mathbf{U}^{t}}^{T} \mathbf{I} \\ \mathbf{R}_{\mathrm{I}} \end{array} \right] \left[ \begin{array}{llllllll} {\mathbf{V}^{t}}^{T} \mathbf{{\Delta}}^{T} \\ \mathbf{R}_{\mathrm{{\Delta}}} \end{array} \right]^{T} \right) [ \mathbf{V}^{t} \quad \mathbf{Q}_{\mathrm{{\Delta}}} ]^{T} \end{array}. $$

(20)

Let $\mathbf {{\Omega }}$ be the middle matrix $\left [ \begin {array}{llllllll} \mathbf {{\Sigma }}^{t} & \mathbf {0} \\ \mathbf {0} & \mathbf {0} \end {array} \right ] + \left [ \begin {array}{llllllll} {\mathbf {U}^{t}}^{T} \mathbf {I} \\ \mathbf {R}_{\mathbf {I}} \end {array} \right ] \left [ \begin {array}{llllllll} {\mathbf {V}^{t}}^{T} \mathbf {{\Delta }}^{T} \\ \mathbf {R}_{\mathrm {{\Delta }}} \end {array} \right ]^{T}$ of (20). To get the incremental SVD of B^t+ 1, we only need to decompose $\mathbf {{\Omega }}$ efficiently. Since $\mathbf {U}^{t}$ is the LSA of B^t, $\mathbf {U}^{t} {\mathbf {U}^{t}}^{T}$ is $\left [ \begin {array}{llllllll} \mathbf {I}_{m} & \mathbf {0} \\ \mathbf {0} & \mathbf {0} \end {array} \right ]_{m^{\prime }}$ . Thus, Q_I is $\left [ \begin {array}{llllllll} \mathbf {0} \\ \mathbf {I}_{m^{\prime }-m} \end {array} \right ]$ and R_I is $[ \mathbf {0} \quad \mathbf {I}_{m^{\prime }-m} ]$. Similarly, since the $\mathbf {V}^{t}_{n^{\prime } \times n}$ of (20) is $[ \mathbf {V}^{t}_{n \times n} \quad \mathbf {0}_{(n^{\prime }-n) \times n} ]$, ${\mathbf {V}^{t}}^{T} \mathbf {{\Delta }}^{T}$ can be decomposed as $[ \mathbf {0} \quad {\mathbf {V}^{t}}^{T} \mathbf {{\Delta }}^{T}_{2} ]$. Hence, $\mathbf {{\Omega }}$ can be rewritten as:

$$ \begin{array}{lllll} \mathbf{{\Omega}} & = \left[ \begin{array}{llllllll} \mathbf{{\Sigma}}^{t} & \mathbf{0} \\ \mathbf{0} & \mathbf{0} \end{array} \right] + \left[ \begin{array}{llllllll} {\mathbf{U}^{t}}^{T} & \mathbf{0} \\ \mathbf{0} & \mathbf{I} \end{array} \right] \left[ \begin{array}{llllllll} \mathbf{0} & {\mathbf{V}^{t}}^{T} \mathbf{{\Delta}}^{T}_{2} \\ \mathbf{R}_{\mathrm{{\Delta}}1} & \mathbf{R}_{\mathrm{{\Delta}}2} \end{array} \right]^{T} \\ & = \left[ \begin{array}{llllllll} \mathbf{{\Sigma}}^{t} & {\mathbf{U}^{t}}^{T} \mathbf{R}^{T}_{\mathrm{{\Delta}}1} \\ \mathbf{{\Delta}}_{2} {\mathbf{V}^{t}} & \mathbf{R}^{T}_{\mathrm{{\Delta}}2} \end{array} \right] \\ s.t. & \quad \mathbf{R}_{\mathrm{{\Delta}}} = [\mathbf{R}_{\mathrm{{\Delta}}1} \quad \mathbf{R}_{\mathrm{{\Delta}}2}] \end{array}, $$

(21)

where $\mathbf {R}_{\mathrm {{\Delta }}1} \!\in \! \mathbb {R}^{(n^{\prime }-n) \times m}$, $\mathbf {R}_{\mathrm {{\Delta }}2} \!\in \! \mathbb {R}^{(n^{\prime }-n) \times (m^{\prime }-m)}$. Thus, we have to get the SVD of $\mathbf {{\Omega }}$ efficiently.

Second, to perform a fast SVD of $\mathbf {{\Omega }}$, we follow the principle of quasi-Gram-Schmidt [34], by approximating $\mathbf {{\Omega }}$ with a reduced matrix $\mathbf {\widehat {{\Omega }}} = \mathbf {C} \mathbf {H} \mathbf {R}^{T}$ where $\mathbf {C}$, $\mathbf {R}$ are sampled from the columns or rows of $\mathbf {{\Omega }}$ separately. Generally, new appended and large singular value related rows or columns, will be sampled. The difference between $\mathbf {{\Omega }}$ and $\mathbf {\widehat {{\Omega }}}$ can be minimized, if $\mathbf {H}$ is $(\mathbf {R}^{-1}_{c} \mathbf {R}^{-T}_{c} ) (\mathbf {C}^{T} \mathbf {{\Omega }} \mathbf {R} ) (\mathbf {R}^{-1}_{r} \mathbf {R}^{-T}_{r} )$, where $\mathbf {R}_{c}$, $\mathbf {R}_{r}$ are upper triangular matrices of the QR factorization on $\mathbf {C}$, $\mathbf {R}$ respectively. Let $\mathbf {{\Psi }}$ be the matrix $\mathbf {R}^{-T}_{c} \mathbf {C}^{T} \mathbf {{\Omega }} \mathbf {R} \mathbf {R}^{-1}_{r}$ and its SVD $\mathbf {U}_{\mathrm {{\Psi }}} \mathbf {{\Sigma }}_{\mathrm {{\Psi }}} \mathbf {V}^{T}_{\mathrm {{\Psi }}}$. Therefore, the SVD of $\mathbf {\widehat {{\Omega }}}$ can be formulated as:

$$ \mathbf{U}_{\widehat{\mathrm{{\Omega}}}} = \mathbf{C} \mathbf{R}^{-1}_{c} \mathbf{U}_{\mathrm{{\Psi}}}, \quad \mathbf{{\Sigma}}_{\widehat{\mathrm{{\Omega}}}} = \mathbf{{\Sigma}}_{\mathrm{{\Psi}}}, \quad \mathbf{V}^{T}_{\widehat{\mathrm{{\Omega}}}} = \mathbf{V}^{T}_{\mathrm{{\Psi}}} \mathbf{R}^{-T}_{r} \mathbf{R}^{T}. $$

(22)

Since the dimension of $\mathbf {{\Psi }}$ is much smaller than that of $\mathbf {\widehat {{\Omega }}}$, its SVD will be very fast. With the fast SVD of $\mathbf {\widehat {{\Omega }}}$, the incremental LSA of $\mathbf {B}^{t + 1}$ can finally be approximated by:

$$ \begin{array}{lllll} \mathbf{B}^{t + 1} & = \mathbf{U}^{t + 1} \mathbf{{\Sigma}}^{t + 1} {\mathbf{V}^{t + 1}}^{T} \\ & = ([ \mathbf{U}^{t} \quad \mathbf{I} ] \mathbf{C} \mathbf{R}^{-1}_{c} \mathbf{U}_{\mathrm{{\Psi}}} ) \mathbf{{\Sigma}}_{\mathrm{{\Psi}}} ([ \mathbf{V}^{t} \mathbf{\quad Q}_{\mathrm{{\Delta}}} ] \mathbf{R} \mathbf{R}^{-1}_{r} \mathbf{V}_{\mathrm{{\Psi}}})^{T} \end{array}, $$

(23)

where $\mathbf {U}^{t + 1} \mathbf {{\Sigma }}^{t + 1} {\mathbf {V}^{t + 1}}^{T}$ is the LSA of $\mathbf {B}^{t + 1}$. Consequently, the LSA of $\mathbf {B}^{t + 1}$ is specified by that of $\mathbf {B}^{t}$ (U^tΣ^tV^tT), as well as some other matrices derived from $\mathbf {U}^{t}$, $\mathbf {{\Sigma }}^{t}$, $\mathbf {V}^{t}$, $\mathbf {{\Delta }}$. With the LSA of $\mathbf {B}^{t + 1}$, we can provide the summarization of evolving events easily in Section 3.2.2. For more information about the derivation, please refer to [9] and [34].