Preference Prediction Based on Eye Movement Using Multi-layer Combinatorial Fusion

  • Christina SchweikertEmail author
  • Louis Gobin
  • Shuxiao Xie
  • Shinsuke Shimojo
  • D. Frank Hsu
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11309)


Face image preference is influenced by many factors and can be detected by analyzing eye movement data. When comparing two face images, our gaze shifts within and between the faces. Eye tracking data can give us insights into the cognitive processes involved in forming a preference. In this paper, a gaze tracking dataset is analyzed using three machine learning algorithms (MLA): AdaBoost, Random Forest, and Mixed Group Ranks (MGR) as well as a newly developed machine learning framework called Multi-Layer Combinatorial Fusion (MCF) to predict a subject’s face image preference. Attributes constructed from the dataset are treated as input scoring systems. MCF involves a series of layers that consist of expansion and reduction processes. The expansion process involves performing exhaustive score and rank combinations, while the reduction process uses performance and diversity to select a subset of systems that will be passed onto the next layer of analysis. Performance and cognitive diversity are used in weighted scoring system combinations and system selection. The results outperform the Mixed Group Ranks algorithm, as well as our previous work using pairwise scoring system combinations.


Combinatorial Fusion Analysis (CFA) Multi-layer Combinatorial Fusion (MCF) Cognitive diversity Machine learning Preference detection 


  1. 1.
    Breiman, L.: Random forests. Mach. Learn. 45, 5–32 (2001)CrossRefGoogle Scholar
  2. 2.
    Chuk, T., Chan, A.B., Shimojo, S., Hsiao, J.H.: Mind reading: discovering individual preferences from eye movements using switching hidden Markov models. In: Papafragou, A., Grodner, D., Mirman, D., Trueswell, J.C. (eds.) Proceedings of the 38th Annual Conference of the Cognitive Science Society, p. 182. Cognitive Science Society, Austin (2016)Google Scholar
  3. 3.
    Fantz, R.L.: The origin of form perception. Sci. Am. 204, 66–72 (1961)CrossRefGoogle Scholar
  4. 4.
    Freund, Y., Schapire, R.E.: A decision-theoretic generalization of on-line learning and an application to boosting. J. Comput. Syst. Sci. 55, 119–139 (1997)MathSciNetCrossRefGoogle Scholar
  5. 5.
    Han, J., Kamber, M., Pei, J.: Data Mining: Concepts and Techniques, 3rd edn. Morgan Kaufmann, San Francisco (2012)zbMATHGoogle Scholar
  6. 6.
    Ho, T.K., et al.: Decision combination in multiple classifier systems. IEEE Trans PAMI 16(1), 66–75 (1994)CrossRefGoogle Scholar
  7. 7.
    Hsu, D.F., Chung, Y.S., Kristal, B.S.: Combinatorial fusion analysis: methods and practice of combining multiple scoring systems. In: Hsu, H.H. (ed.) Advanced Data Mining Technologies in Bioinformatics, pp. 1157–1181. Idea Group Inc., Calgary (2006)Google Scholar
  8. 8.
    Hsu, D.F., Kristal, B.S., Schweikert, C.: Rank-score characteristics (RSC) function and cognitive diversity. In: Yao, Y., Sun, R., Poggio, T., Liu, J., Zhong, N., Huang, J. (eds.) BI 2010. LNCS (LNAI), vol. 6334, pp. 42–54. Springer, Heidelberg (2010). Scholar
  9. 9.
    Hsu, D.F., Taksa, I.: Comparing rank and score combination methods for data fusion in information retrieval. Inf. Retr. 8(3), 449–480 (2005)CrossRefGoogle Scholar
  10. 10.
    Khushaba, R., et al.: Consumer neuroscience: assessing the brain response to marketing stimuli using electroencephalogram (EEG) and eye tracking. Expert Syst. Appl. 40(9), 3803–3812 (2013)CrossRefGoogle Scholar
  11. 11.
    König, P., et al.: Eye movements as a window to cognitive processes. J. Eye Mov. Res. 9(5), 1–16 (2016)Google Scholar
  12. 12.
    Kustiawan, I., Liu, C.-Y., Hsu, D.F.: Vertical Handoff decision using fuzzification and combinatorial fusion. IEEE Commun. Lett. 21, 2089–2092 (2017)CrossRefGoogle Scholar
  13. 13.
    Li, Y., Hsu, D.F., Chung, S.M.: Combination of multiple feature selection methods for text categorization by using combinatorial fusion analysis and rank-score characteristic. Int. J. Artif. Intell. Tools 22(2), 1350001 (2013)CrossRefGoogle Scholar
  14. 14.
    Liao, H.I., Shimojo, S.: Dynamic preference formation via gaze and memory. In: Sharot, T., Dolan, R. (ed.) Neuroscience of Preference and Choice, pp. 277–292. Elsevier Inc. (2012)Google Scholar
  15. 15.
    Liao, H.I., Yeh, S.L., Shimojo, S.: Novelty vs. familiarity principles in preference decisions: task-context of past experience matters. Front. Psychol. 2(43), 1–8 (2011)Google Scholar
  16. 16.
    Lin, K.L., et al.: Feature selection and combination criteria for improving accuracy in protein structure prediction. IEEE Trans. Nanobiosci. 6(2), 186–196 (2007)CrossRefGoogle Scholar
  17. 17.
    Little, A.C., Jones, B.C., DeBruine, L.M.: Facial attractiveness: evolutionary based research. Philos. Trans. R. Soc. B: Biol. Sci. 366(1571), 1638–1659 (2011)CrossRefGoogle Scholar
  18. 18.
    Lyons, D.M., Hsu, D.F.: Combining multiple scoring systems for target tracking using rank score characteristics. Inform. Fus. 10(2), 124–136 (2009)CrossRefGoogle Scholar
  19. 19.
    Mandler, G., Nakamura, Y., Van Zandt, B.J.: Exp. Psychol. Learn. Mem. Cogn. 13, 646–648 (1987)CrossRefGoogle Scholar
  20. 20.
    Melnik, O., Vardi, Y., Zhang, C.H.: Mixed group ranks: preference and confidence in classifier combination. IEEE Trans. Pattern Anal. Mach. Intell. 26(8), 973–981 (2004)CrossRefGoogle Scholar
  21. 21.
    Raney, G.E., Campbell, S.J., Bovee, J.C.: Using eye movements to evaluate the cognitive processes involved in text comprehension. J. Vis. Exp. JoVE 83, 50780 (2014)Google Scholar
  22. 22.
    Schweikert, C., Shimojo, S., Hsu, D.F.: Detecting preferences based on eye movement using combinatorial fusion. In: Wang, Y., et al. (ed.) Proceedings of the 2016 IEEE 15th International Conference on Cognitive Informatics & Cognitive Computing (ICCI*CC 2016), pp. 336–343 (2016)Google Scholar
  23. 23.
    Schweikert, C., Brown, S., Tang, Z., Smith, P.R., Hsu, D.F.: Combining multiple ChIP-seq peak detection systems using combinatorial fusion. BMC Genom. 13(Suppl 8), S12 (2012)CrossRefGoogle Scholar
  24. 24.
    Shimojo, S., Simion, C., Changizi, M.: Gaze and preference – orienting behavior as a somatic precursor of preference decision. In: Adams, Jr., R.B., Ambady, N., Nakayama, K., Shimojo, S. (eds.) Social Vision, pp. 151–163. Oxford Univ. Press (2011)Google Scholar
  25. 25.
    Shimojo, S., Simion, C., Shimojo, E., Scheier, C.: Gaze bias both reflects and influences preference. Nat. Neurosci. 6, 1317–1322 (2003)CrossRefGoogle Scholar
  26. 26.
    Wang, Y., Wang, Y.: Cognitive informatics models of the brain. IEEE Trans. Syst. Man Cybern. Part C (Appl. Rev.) 36(2), 203–207 (2006)MathSciNetCrossRefGoogle Scholar
  27. 27.
    Yang, J.M., et al.: Consensus scoring for improving enrichment in virtual screening. J. Chem. Inform. Model. 45, 1134–1146 (2005)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Christina Schweikert
    • 1
    Email author
  • Louis Gobin
    • 2
  • Shuxiao Xie
    • 2
  • Shinsuke Shimojo
    • 3
  • D. Frank Hsu
    • 2
  1. 1.Division of Computer Science, Mathematics and ScienceSt. John’s UniversityQueensUSA
  2. 2.Laboratory of Informatics and Data Mining, Department of Computer and Information ScienceFordham UniversityNew YorkUSA
  3. 3.Division of Biology and Biological Engineering/Computation and Neural SystemsCalifornia Institute of TechnologyPasadenaUSA

Personalised recommendations