Interpretation of text patterns

Abstract

Patterns are used as a fundamental means to analyse data in many text mining applications. Many efficient techniques have been developed to discover patterns. However, the excessive number of discovered patterns and lack of grounded (e.g. a priori defined) semantics have made it difficult for a user to interpret and explore the patterns. An insight into the meanings of the patterns can benefit users in the process of exploring them. In this regard, this paper presents a model to automatically interpret patterns by achieving two goals: (1) providing the meanings of patterns in terms of ontology concepts and (2) providing a new method for generating and extracting features from an ontology to describe the relevant information more effectively. Taking advantage of a domain ontology and a set of relevant statistics (e.g. term frequency in a document, inverse term frequency in a domain ontology, etc.), our proposed model can give an insight into the hidden meanings of the patterns. The model is evaluated by comparing it with different baseline models on three standard datasets. The results show that the performance of the proposed model is significantly better than baseline models.

Appendix A: Description of evaluation measures

For a given topic, recall is the fraction of relevant documents that are retrieved, i.e.

$$\begin{aligned} r_{c} = \frac{\left| \left\{ rl \right\} \cap \left\{ rt \right\} \right| }{\left| \left\{ rl \right\} \right| }; \end{aligned}$$

precision is the fraction of retrieved documents that are relevant, i.e.

$$\begin{aligned} p_{c} = \frac{\left| \left\{ rl \right\} \cap \left\{ rt \right\} \right| }{\left| \left\{ rt \right\} \right| }; \end{aligned}$$

where rl is a relevant document rt is a retrieved document.

We want both the precision and the recall to be high, rather than the precision being high but the recall low or vice versa. To measure this property $F_{score}$ is used. It is defined by the following formula:

$$\begin{aligned} F_{score} = (1+\sigma ^{2})\frac{p_{c}\times r_{c}}{\sigma ^{2}p_{c}+r_{c}}. \end{aligned}$$

The $\sigma$ is a user defined-value that reflects our concern about false negative (irrelevant) versus false positive (relevant), which is conventionally assigned to 1 (in that case it is called $F_{1}$). The $F_{score}$ is the harmonic mean of recall and precision. The harmonic mean tends to be closer to the smaller of the two values. Therefore, $F_{score}$ will be high when both of recall and precision are high. The break-even point (BP) is the value for which both recall and precision are equal.

Precision, recall, F measure and break-even point are set-based measures that are computed based on unordered sets of documents. We can also use measures that evaluate the ranked (ordered) documents, which are now standard in information filtering systems. All the retrieved documents are taken into account in the precision calculation. However, in a ranked document context, it can be evaluated at a given cutoff that considers only the topmost results returned by the system. This measure is called $top-u$ precision, in our evaluation, we use the top-20 precision.

For each $top-u$ returned documents, precision and recall values can be plotted on a precision-recall curve. If the $(u+1)$th document returned by the system is nonrelevant then recall is the same as for the top u documents, but precision is dropped. If $(u+1)$th document is relevant, then both precision and recall increase. This curve has a saw-tooth shape, but often jiggles are removed by interpolated precision. The interpolated precision $p_{int}$ at a certain recall level $r_c$ is defined as the highest precision found for any recall level $r'_c \ge r_c$, i.e. $p_{int} (r_c) = \max _{r'_c \ge r_c} p_c(r'_c)$, where $p_{int} (r_c)$ is interpolated precision at recall level $r_c$, $p_c(r'_c)$ is precision at recall level $r'_c$. By definition, the interpolated precision at a recall of 0 is 1.

Mean Average Precision (MAP) is mean for average precision. Let, we are given some topics; and for each topic, we are given some documents sorted according to their relevance to the topic. The equation for calculating the AP (average precision) for a filtering system that returns u documents sorted according to their relevance to a topic is:

$$\begin{aligned} AP = \frac{\sum _{i=1}^{u} (p_{c_i})\times (r_{v_i})}{\left| \left\{ rl \right\} \right| }, \end{aligned}$$

where $p_{c_i}$ = $p_c$ at ith position and $r_{v_i}$ is the relevance value (i.e. 0 or 1) of the document in the ith position of the sorted list. The MAP is an average of APs over all the topics. MAP is commonly used by TREC participants, and it gives the indication of the order-matters precision.

A $t-Test$ is a parametric statistical hypothesis test, while Wilcoxon signed-rank test is a non-parametric statistical hypothesis test. These two tests are used to determine if two sets of data are significantly different from each other. Unlike parametric statistics, nonparametric statistics do not assume any specific probability distributions of the variables being assessed. We apply both of these to statistically analyse the difference between our proposed model and the best baseline model’s results—for the measure Top-20, $F_{1}$, BP and MAP.