Text Processing

Text provides crucial cues for understanding content. For example, the closed captions in broadcast television programs and subtitles in DVD movies facilitate video consumption for viewers. When a transcript is not available for certain content, automatic speech recognition can be used to extract linguistic information. Text information is much more concise than corresponding audio or video. The reason is that we need language knowledge to understand text, and the knowledge itself does not need to be embedded in the text data. For example, we only need five characters to express a “plane,” but to show a video clip of plane takes millions of bytes. Text streams contain very rich semantic information. How to effectively extract information from text is an important component in video content analysis.

In this chapter, we will introduce some fundamentals in text processing that are relevant to content analysis, information extraction, and information retrieval. Specifically, we will discuss part of speech tagging, named entity extraction, text capitalization, stemming, term weighting, and document ranking. We will also present a few methods for story segmentation and text summarization.