Automatic Indexing: The Selection of Natural Language Index Terms

Conclusions

Many of the techniques for selecting natural language index terms from texts rely upon simple assumptions about distribution patterns of individual words. In some cases the methods bear upon linguistic knowledge regarding a micro level of text description, i.e., bearing upon the vocabulary, syntax, and semantics of the individual sentences, clauses, and phrases. The linguistic knowledge is involved in stemming procedures, phrase recognition, and phrase normalization. The existing techniques were originally developed to index heterogeneous document collections, which explains the rather shallow approach. There is a growing interest to incorporate knowledge regarding a macro level of text description into the indexing systems. This knowledge can not only be incorporated as heuristics in the search for good terms, but can also be the basis of probabilistic term distributions that are useful in indexing.

There are of course the problems of indexing with natural language terms discussed in chapter 3 including synonymy, homonymy, and polysemy and the set of terms being an unordered set of phrases or individual words. In the next chapters of this part, alternative text indexing and abstracting techniques are described that alleviate these problems.