Hierarchical Learning of Dependent Concepts for Human Activity Recognition

Abstract

In multi-class classification tasks, like human activity recognition, it is often assumed that classes are separable. In real applications, this assumption becomes strong and generates inconsistencies. Besides, the most commonly used approach is to learn classes one-by-one against the others. This computational simplification principle introduces strong inductive biases on the learned theories. In fact, the natural connections among some classes, and not others, deserve to be taken into account. In this paper, we show that the organization of overlapping classes (multiple inheritances) into hierarchies considerably improves classification performances. This is particularly true in the case of activity recognition tasks featured in the SHL dataset. After theoretically showing the exponential complexity of possible class hierarchies, we propose an approach based on transfer affinity among the classes to determine an optimal hierarchy for the learning process. Extensive experiments show improved performances and a reduction in the number of examples needed to learn.

Appendices

Appendix A

Proof

Theorem 1. It can be explained by observing that, for $K+1$ concepts containing K existed concepts $c_1, \cdots c_K$ and a new added concept $\gamma$, we can produce the first level trees combinations as below. Notice that each atomic element o can be one of the $c_1, \cdots c_K$ concepts. In order to compute the total number of trees combinations, we show what is the number of tree combinations by assigning the K concepts to each item:

• $(\gamma (\overbrace{o\cdots o}^{K \text {concepts}}))$: the number of trees combinations by taking the concept labels into the account are: ${K \atopwithdelims ()0} L(1) \times 2 \times L(K)$; the reason for multiplying the number of trees combinations for K concepts to 2 is because while the left side contains an atomic $\gamma$ concept, there are two choices for the right side of the tree in the first level: either we compute the total number of trees for K concepts from the first level or we keep the first level as a $\overbrace{o\cdots o}^{K \text {concepts}}$ atomics and keep all K concepts together, then continue the number of K trees combinations from the second level of the tree.

• $((\gamma o)(\overbrace{o\cdots o}^{K-1 \text {concepts}}))$: similar to the previous part we have ${K \atopwithdelims ()1} L(2) \times 2 \times L(K-1)$ trees combinations by taking the concepts labels into the account. ${K \atopwithdelims ()1}$ indicates the number of combinations for choosing a concept from the K concept and put it with the new concept separately. While L(2) is the number of trees combinations for the left side of tree separated with the new concept $\gamma$.

• $((\gamma oo)(\overbrace{o\cdots o}^{K-2 \text {concepts}}))$, $\cdots$

• $((\gamma \overbrace{o\cdots o}^{K-1 \text {concepts}})o)$: ${K \atopwithdelims ()K-1} L(K) L(1)$ in this special part, we follow the same formula except the single concept in the right side has only one possible combination in the first level equal to L(1).

All in all, the sum of these items calculates the total number of tree hierarchies for $K+1$ concepts.

The first few number of total number of trees combinations for 1, 2, 3, 4, 5, 6, $7, 8, 9,10, \cdots$ concepts are: 1,  1,  4,  26,  236,  2752,  39208,  660032,  12818912,   $282137824,\cdots$. In the case of the SHL dataset that we use in the empirical evaluation, we have 8 different concepts and thus, the number of different types of hierarchies for this case is $L(8) = 660,032$.

Appendix B Training Details

We use Tensorflow for building the encoders/decoders. We construct encoders by stacking Conv1d/ReLU/MaxPool blocks. These blocks are followed by a Fully Connected/ReLU layers. Encoders performance estimation is based on the validation loss and is framed as a sequence classification problem. As a preprocessing step, annotated input streams from the huge SHL dataset are segmented into sequences of 6000 samples which correspond to a duration of 1 min. given a sampling rate 100 Hz. For weight optimization, we use stochastic gradient descent with Nesterov momentum of 0.9 and a learning-rate of 0.1 for a minimum of 12 epochs (we stop training if there is no improvement). Weight decay is set to 0.0001. Furthermore, to make the neural networks more stable, we use batch normalization on top of each convolutional layer. We use SVMs as our ERMs in the derived hierarchies.

Appendix C Evaluation Metrics

In hierarchical classification settings, the hierarchical structure is important and should be taken into account during model evaluation [17]. Various measures that account for the hierarchical structure of the learning process have been studied in the literature. They can be categorized into: distance-based; depth-dependent; semantics-based; and hierarchy-based measures. Each one is displaying advantages and disadvantages depending on the characteristics of the considered structure [5]. In our experiments, we use the H-loss, a hierarchy-based measure defined in [3]. This measure captures the intuition that “whenever a classification mistake is made on a node of the taxonomy, then no loss should be charged for any additional mistake occurring in the sub-tree of that node.” $\ell _H(\hat{y}, y) = \sum _{i=1}^{N} \{ \hat{y}_i \ne y_i \wedge \hat{y}_j = y_j, j \in Anc(i) \}$, where $\hat{y} = (\hat{y}_1, \cdots \hat{y}_N)$ is the predicted labels, $y = (y_1, \cdots y_N)$ is the true labels, and Anc(i) is the set of ancestors for the node i.