“I Know It When I See It”: How Experts and Novices Recognize Good Design

  • Kesler TannerEmail author
  • James Landay
Part of the Understanding Innovation book series (UNDINNO)


Design novices have limited design experience and typically lack the skills or confidence to create good design, however, they may be able to recognize good design. To assess this ability, 53 novice designers and 52 expert designers participated in an online study where they evaluated a series of websites based on aesthetic appeal using two different modes of comparison. Results show that both experts and novices are able to recognize good design and that novices are able to do so almost as well as experts (76.5% accuracy compared to 81.2%). The greatest determinant of whether a participant would correctly identify a higher-rated design was the difference in the two websites’ ground-truth aesthetic ratings. However, expertise and the mode by which the comparison was presented had a significant impact on accuracy (Keep-the-Best = 83.6% and Tournament = 74.1%).


Novice designers Comparative design Design subjectivity Design expertise Visual design Web design 



We’d like to thank Jacob O. Wobbrock, Rob Semmens, Kurt Gee and Roy Roy for their assistance with statistics. Any opinions, findings, conclusions or recommendations expressed in this work are those of the authors and do not necessarily reflect those of any supporter listed above.


  1. Bogumil, R. J. (1985). The reflective practitioner: How professionals think in action. Proceedings of the IEEE [Internet]. [cited 2017 Sep 15], 73(4), 845–846. Available from: Scholar
  2. Buxton, B. (2010). Sketching user experiences: Getting the design right and the right design [Internet]. [cited 2017 Sep 15]. Available from:
  3. Christiaans, H. H. C. M., & Dorst, K. H. (1992). Cognitive models in industrial design engineering: A protocol study. American Society of Mechanical Engineers Design Engineering Division.Google Scholar
  4. Cross, N. (2004). Expertise in design: An overview. Design Studies. [Internet]. Elsevier; Sep 1 [cited 2017 Sep 15];25(5), 427–441. Available from: Scholar
  5. Dow, S., & Klemmer, S. R. (2010). The efficacy of prototyping under time constraints. Design Thinking. [Internet]. [cited 2017 Sep 15], pp. 111–128. Available from: Scholar
  6. Dow, S. P., Glassco, A., Kass, J., Schwarz, M., Schwartz, D. L., & Klemmer, S. R. (2012). Parallel prototyping leads to better design results, more divergence, and increased self-efficacy. Design Thinking Research Studies. Co-Creation Practice [Internet]. [cited 2017 Sep 15], pp. 127–153. Available from: Scholar
  7. Faste, R. (1995). The role of aesthetics in engineering. Japan Society of Mechanical Engineering Journal. [Internet]. [cited 2017 Sep 15]. Available from:
  8. Faste, H. (2017). Intuition in design: Reflections on the iterative aesthetics of form. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems – CHI EA ’17 [Internet]. [cited 2017 Sep 15], pp. 3403–3413. Available from:
  9. Hartmann, B., Klemmer, S., Bernstein, M., Abdulla, L., Burr, B., Robinson-Mosher, A., et al. (2006) Reflective physical prototyping through integrated design, test, and analysis. In Proceedings of 19th Annual ACM Symposium on User interface Software Technology [Internet]. [cited 2017 Sep 15], pp. 299–308. Available from:
  10. Hassenzahl, M. (2004, December 1). The interplay of beauty, goodness, and usability in interactive products. Human-Computer Interactions. [Internet]. L. Erlbaum Associates [cited 2017 Sep 15], 19(4), 319–349. Available from: Scholar
  11. Kulkarni, C., Wei, K. P., Le, H., Chia, D., Papadopoulos, K., & Cheng, J., et al. (2013). Peer and self assessment in massive online classes. ACM Transactions on Computer-Human Interaction. [Internet]. [cited 2017 Sep 15], 20(6), 1–31. Available from: Scholar
  12. Kurosu, M., & Kashimura, K. (1995). Creativity of. Apparent usability vs. inherent usability: Experimental analysis on the determinants of the apparent usability. In Proceedings of ACM Conference on Human Factors in Computer Systems [Internet]. New York, New York, USA: ACM Press; [cited 2017 Sep 15], 292–293. Available from:
  13. Lloyd, P., & Scott, P. (1994). Discovering the design problem. Design Studies. [Internet]. [cited 2017 Sep 15], 15(2), 125–140. Available from: Scholar
  14. Luther, K., Pavel, A., Wu, W., Tolentino, J., Agrawala, M., & Hartmann, B., et al. (2014) CrowdCrit: Crowdsourcing and aggregating visual design critique. In Proceedings of Companion Publ. 17th ACM Conference on Computer Supported Cooperative Work and Social Computing – CSCW Companion ’14 [Internet]. [cited 2017 Sep 15], pp. 21–24. Available from:
  15. Luther, K., Tolentino, J., Wu, W., Pavel, A., Bailey, B. P., Agrawala, M., et al. (2015) Structuring, aggregating, and evaluating crowdsourced design critique. In Proceedings of 18th ACM Conference on Computer Supported Cooperative Work and Social Computing – CSCW ’15 [Internet]. [cited 2017 Sep 15], pp. 473–485. Available from:
  16. Marks, J., Ruml, W., Ryall, K., Seims, J., Shieber, S., & Andalman, B., et al. (1997). Design galleries: a general approach to setting parameters for computer graphics and animation. In Proceedings of 24th Annual Conference on Computer Graphics Interactive Technology – SIGGRAPH ’97 [Internet]. [cited 2017 Sep 15], pp. 389–400. Available from:
  17. O’Donovan, P., Agarwala, A., & Hertzmann, A. (2015). DesignScape: Design with interactive layout suggestions. In Proceedings of 33rd Annual Conference on Human Factors in Computer Systems – CHI ’15 [Internet]. [cited 2017 Sep 15], pp. 1221–1224. Available from:
  18. Petitmengin-peugeot, C. (1999). The intuitive experience. Journal of Consciousness Studies. [Internet]. [cited 2017 Sep 15], 2, 43–77. Available from:
  19. Reinecke, K., & Gajos, K. Z. (2014). Quantifying visual preferences around the world. In Proceedings of 32rd Annual ACM Conference on Human Factors in Computer Systems – CHI ’14 [Internet]. ACM Press, New York, USA; [cited 2017 Sep 15], pp. 11–20. Available from:
  20. Salter, A., & Whyte, J. (2004). Serious play: How the world’s best companies simulate to innovate [Internet]. Technovation. [cited 2017 Sep 15], pp. 277–278. Available from:’s+Best+Companies+Simulate+to+Innovate&ots=RFdJu21jIC&sig=rHiLvl2c91_MH8Zao_nEZmabVhU
  21. Tidwell, J. (2005). Designing interfaces: Patterns for effective interaction design [Internet]. OReilly Media Inc. [cited 2017 Sep 19]. p. 352. Available from:
  22. Todi, K., Weir, D., & Oulasvirta, A. (2016). Sketchplore: Sketch and explore with a layout optimiser. In Proceedings of 2016 ACM Conference [Internet]. [cited 2017 Sep 15]; Available from:
  23. Tohidi, M., Buxton, W., Baecker, R., & Sellen, A. (2006). Getting the right design and the design right: Testing many is better than one. In Proceedings of CHI 2006 Conference on Human Factors on Computer Systems [Internet]. [cited 2017 Sep 15], pp. 1243–1252. Available from:
  24. Van Duyne, D. K., Landay, J. A., & Hong, J. I. (2007). The design of sites: Patterns for creating winning web sites [Internet]. [cited 2017 Sep 19]. Available from:
  25. Willett, W., Heer, J., & Agrawala, M. (2012). Strategies for crowdsourcing social data analysis. In Proceedings of 2012 ACM Annual Conference on Human Factors on Computer Systems – CHI ’12 [Internet]. [cited 2017 Sep 15], p. 227. Available from:
  26. Xu, A., & Bailey, B. P. (2012) What do you think? A case study of benefit, expectation, and interaction in a large online critique community. In Proceedings of 15th ACM Conference on Computer Supported Cooperative Work and Social Computing – CSCW’12 [Internet]. [cited 2017 Sep 15], 295. Available from:
  27. Xu, A., Huang, S.-W., & Bailey, B. (2014). Voyant: generating structured feedback on visual designs using a crowd of non-experts. In Proceedings of 17th ACM Conference on Computer Supported Cooperative Work and Social Computing – CSCW ’14 [Internet]. [cited 2017 Sep 15], pp. 1433–1444. Available from:
  28. Xu, A., Rao, H., Dow, S. P., & Bailey, B. P. (2015). A classroom study of using crowd feedback in the iterative design process. In Proceedings of 18th ACM Conference on Computer Supported Cooperative Work and Social Computing – CSCW ’15 [Internet]. [cited 2017 Sep 15], pp. 1637–1648. Available from:
  29. Yuan, A., Luther, K., Krause, M., Vennix, S. I., Dow, S. P., & Hartmann, B. (2016). Almost an expert: The effects of rubrics and expertise on perceived value of crowdsourced design critiques. In Proceedings of 19th ACM Conference on Computer Supported Cooperative Work and Social Computing – CSCW ’16 [Internet]. [cited 2017 Sep 15], pp. 1003–1015. Available from:

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Stanford UniversityStanfordUSA

Personalised recommendations