Solutions for Complex, Multi Data Type and Multi Tool Analysis: Principles and Applications of Using Workflow and Pipelining Methods

Abstract

Analytical workflow technology, sometimes also called data pipelining, is the fundamental component that provides the scalable analytical middleware that can be used to enable the rapid building and deployment of an analytical application. Analytical workflows enable researchers, analysts and informaticians to

• integrate and access data and tools from structured and non-structured data sources so that analytics can bridge different silos of information;

• compose multiple analytical methods and data transformations without coding;

• rapidly develop applications and solutions by visually constructing analytical workflows that are easy to revise should the requirements change;

• access domain-specific extensions for specific projects or areas, for example, text extraction, visualisation, reporting, genetics, cheminformatics, bioinformatics and patient-based analytics;

• automatically deploy workflows directly into web portals and as web services to be part of a service-oriented architecture (SOA).

By performing workflow building, using a middleware layer for data integration, it is a relatively simple exercise to visually design an analytical process for data analysis and then publish this as a service to a web browser. All this is encapsulated into what can be referred to as an ‘Embedded Analytics’ methodology which will be described here with examples covering different scientifically focused data analysis problems.