Testing and Best Practices

Abstract

When we process information to identify relations or extract entities, to type or classify them, or to fill out their attributes, we need to gauge how well our algorithms work. KM poses a couple of differences from traditional scientific hypothesis testing. The problems we are dealing with in information retrieval (IR), natural language understanding or processing (NLP), and machine learning (ML) are all statistical classification problems, specifically in binary classification. The most common scoring method to gauge the ‘accuracy’ of these classification problems uses statistical tests based on two metrics: negatives or positives, and true or false. We discuss a variety of statistical tests using the four possible results from these metrics (e.g., false positive). Testing scripts range from standard unit tests applied against platform tools to ones that do coherency and consistency checks across the knowledge structure or create reference standards for machine learning or inform improvements. We offer best practices learned from client deployments in areas such as data treatment and dataset management, creating and using knowledge structures, and testing, analysis, and documentation. Modularity in knowledge graphs, or consistent attention to UTF-8 encoding in data structures, or emphasis on ‘semi-automatic’ approaches, or use of literate programming and notebooks to record tests and procedures are just a few of the examples where lines blur between standard and best practices. Finding ways to identify and agree upon shared vocabularies and understandings is a central task of modeling the domain, and it involves practices in collaboration, naming, and use of these knowledge structures.