Automatically Defined Groups for Knowledge Acquisition from Computer Logs and Its Extension for Adaptive Agent Size

Recently, a large amount of data is stored in databases through the advance of computer and network environments. To acquire knowledge from the databases is important for analyses of the present condition of the systems and for predictions of coming incidents. The log file is one of the databases stored automatically in computer systems. Unexpected incidents such as system troubles as well as the histories of daily service programs' actions are recorded in the log files. System administrators have to check the messages in the log files in order to analyze the present condition of the systems. However, the descriptions of the messages are written in various formats according to the kinds of service programs and application software. It may be difficult to understand the meaning of the messages without the manuals or specifications. Moreover, the log files become enormous, and important messages are liable to mingle with a lot of insignificant messages. Therefore, checking the log files is a troublesome task for administrators.

Log monitoring tools such as SWATCH [1], in which regular expressions for representing problematic phrases are used for pattern matching, are effective for detecting well-known typical error messages. However, various programs running in the systems may be open source software or software companies’ products, and they may have been newly developed or upgraded recently. Therefore, it is impossible to detect all the problematic messages by the predefined rules. In addition, in order to cope with illegal use by hackers, it is important to detect unusual behavior such as the start of the unsupposed service program, even if the message does not correspond to the error message. To realize this system, the error-detection rules depending on the environment of the systems should be acquired adaptively by means of evolution or learning.