Automated Supervision of Data Production – Managing the Creation of Statistical Reports on Periodic Data

Abstract

Data production systems are generally very large, distributed and complex systems used for creating advanced (mainly statistical) reports. Typically, data is gathered periodically and then subsequently aggregated and separated during numerous production steps. These production steps are arranged in a specific sequence (workflow or production chain), and can be located worldwide. Today, a need for improving and automating methods of supervision for data production systems has been recognised. Supervision in this context entails planning, monitoring and controlling data production. Since there are usually alternate solutions, it makes good sense to consider several approaches. The two most significant approaches are introduced here for improving this supervision, the ‘closely coupled -’ and the ‘loosely coupled approach’. In either situation, dates, costs, resources, and system health information is made available to management, production operators and administrators to support a timely and smooth production of periodic data. Both approaches are theoretically described and compared. The main finding is that both are useful, but in different cases. The main advantages of the ‘closely coupled approach’ are the large production optimisation potential and a production overview in the form of a job execution plan, whereas the ‘loosely coupled approach’ mainly supports unhindered job execution without adapting legacy components and offers a sophisticated production overview in form of a milestone schedule. Ideas for further research include investigation of other potential approaches and theoretical and practical comparison.