Computational Statistics

, Volume 30, Issue 2, pp 293–316

Using visual statistical inference to better understand random class separations in high dimension, low sample size data

  • Niladri Roy Chowdhury
  • Dianne Cook
  • Heike Hofmann
  • Mahbubul Majumder
  • Eun-Kyung Lee
  • Amy L. Toth
Original Paper

DOI: 10.1007/s00180-014-0534-x

Cite this article as:
Roy Chowdhury, N., Cook, D., Hofmann, H. et al. Comput Stat (2015) 30: 293. doi:10.1007/s00180-014-0534-x

Abstract

Statistical graphics play an important role in exploratory data analysis, model checking and diagnosis. With high dimensional data, this often means plotting low-dimensional projections, for example, in classification tasks projection pursuit is used to find low-dimensional projections that reveal differences between labelled groups. In many contemporary data sets the number of observations is relatively small compared to the number of variables, which is known as a high dimension low sample size (HDLSS) problem. This paper explores the use of visual inference on understanding low-dimensional pictures of HDLSS data. Visual inference helps to quantify the significance of findings made from graphics. This approach may be helpful to broaden the understanding of issues related to HDLSS data in the data analysis community. Methods are illustrated using data from a published paper, which erroneously found real separation in microarray data, and with a simulation study conducted using Amazon’s Mechanical Turk.

Keywords

Statistical graphics Lineup Visualization Projection pursuit  Data mining 

Copyright information

© Springer-Verlag Berlin Heidelberg 2014

Authors and Affiliations

  • Niladri Roy Chowdhury
    • 1
  • Dianne Cook
    • 1
  • Heike Hofmann
    • 1
  • Mahbubul Majumder
    • 2
  • Eun-Kyung Lee
    • 3
  • Amy L. Toth
    • 4
    • 5
  1. 1.Department of StatisticsIowa State UniversityAmesUSA
  2. 2.Department of MathematicsUniversity of NebraskaOmahaUSA
  3. 3.Department of StatisticsEWHA Womans UniversitySeoulKorea
  4. 4.Department of Ecology, Evolution, and Organismal BiologyIowa State UniversityAmesUSA
  5. 5.Department of EntomologyIowa State UniversityAmesUSA