Abstract
In many areas of non-photorealistic and illustrative rendering, considerable progress has been made toward synthesizing traditional artistic and illustrative techniques. However, evaluation and validation of such images have only been attempted relatively recently. This chapter surveys evaluation approaches that have been applied successfully in non-photorealistic and illustrative rendering. It provides an overview over different evaluation approaches including qualitative and quantitative techniques and gives examples for how to approach evaluation in the NPR context. Collectively, the described techniques do not only answer the question of whether an NPR technique is able to replicate a traditional technique successfully but also what implications the use of NPR techniques has and what opinion people have about different NPR techniques as compared to traditional depictions.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Acevedo, D., Laidlaw, D., Drury, F.: Using visual design expertise to characterize the effectiveness of 2D scientific visualization methods. In: Proceedings Compendium of IEEE InfoVis and Visualization 2005, pp. 111–112 (2005). doi:10.1109/VIS.2005.109
AlMeraj, Z., Kaplan, C.S., Asente, P., Lank, E.: Towards ground truth in geometric textures. In: Proc. NPAR, pp. 17–26. ACM, New York (2011). doi:10.1145/2024676.2024679
Bénard, P., Thollot, J., Sillion, F.: Quality assessment of fractalized NPR textures: a perceptual objective metric. In: Proc. APGV, pp. 117–120. ACM, New York (2009). doi:10.1145/1620993.1621016
Carpendale, S.: Evaluating information visualizations. In: Information Visualization: Human-Centered Issues and Perspectives. LNCS, vol. 4950, pp. 19–45. Springer, Berlin (2008). doi:10.1007/978-3-540-70956-5_2
Cohen, J.: The Earth is round (p<0.05). Am. Psychol. 49(12), 997–1003 (1994). doi: 10.1037/0003-066X.49.12.997
Cole, F.: Line drawings of 3D models. Ph.D. thesis, Princeton University (2009)
Cole, F., Golovinskiy, A., Limpaecher, A., Barros, H.S., Finkelstein, A., Funkhouser, T., Rusinkiewic, S.: Where do people draw lines? ACM Trans. Graph. 27(3), 88 (2008). doi:10.1145/1360612.1360687
Cole, F., Sanik, K., DeCarlo, D., Finkelstein, A., Funkhouser, T., Rusinkiewicz, S., Singh, M.: How well do line drawings depict shape? ACM Trans. Graph. 28(3), 28 (2009). doi:10.1145/1531326.1531334
Cole, F., Golovinskiy, A., Limpaecher, A., Barros, H.S., Finkelstein, A., Funkhouser, T., Rusinkiewicz, S.: Where do people draw lines? Commun. ACM 55(1), 107–115 (2012). doi: 10.1145/2063176.2063202
Duke, D.J., Barnard, P.J., Halper, N., Mellin, M.: Rendering and affect. Comput. Graph. Forum 22(3), 359–368 (2003). doi:10.1111/1467-8659.00683
Field, A., Hole, G.: How to Design and Report Experiments. Sage, London (2003)
Fischer, J., Cunningham, D., Bartz, D., Wallraven, C., Bülthoff, H., Straßer, W.: Measuring the discernability of virtual objects in conventional and stylized augmented reality. In: Proc. EGVE, pp. 53–61. Eurographics Association, Goslar (2006). doi:10.2312/EGVE/EGVE06/053-061
Gatzidis, C., Papakonstantinou, S., Brujic-Okretic, V., Baker, S.: Recent advances in the user evaluation methods and studies of non-photorealistic visualisation and rendering techniques. In: Proc. IV, pp. 475–480. IEEE Comput. Soc., Los Alamitos (2008). doi:10.1109/IV.2008.75
Gerstner, T., DeCarlo, D., Alexa, M., Finkelstein, A., Gingold, Y., Nealen, A.: Pixelated image abstraction. In: Proc. NPAR, pp. 29–36. Eurographics Association, Goslar (2012). doi:10.2312/PE/NPAR/NPAR12/029-036
Girshick, A., Interrante, V., Haker, S., Lemoine, T.: Line direction matters: an argument for the use of principal directions in 3D line drawings. In: Proc. NPAR, pp. 43–52. ACM, New York (2000). doi:10.1145/340916.340922
Gooch, B.: Human facial illustrations: creation and evaluation using behavioral studies and functional magnetic resonance imaging. Ph.D. thesis, University of Utah, USA (2003)
Gooch, A.A., Willemsen, P.: Evaluating space perception in NPR immersive environments. In: Proc. NPAR, pp. 105–110. ACM, New York (2002). doi:10.1145/508530.508549
Gooch, B., Reinhard, E., Gooch, A.A.: Human facial illustrations: creation and psychophysical evaluation. ACM Trans. Graph. 23(1), 27–44 (2004). doi:10.1145/966131.966133
Gooch, A.A., Long, J., Ji, L., Estey, A., Gooch, B.S.: Viewing progress in non-photorealistic rendering through Heinlein’s lens. In: Proc. NPAR, pp. 165–171. ACM, New York (2010). doi:10.1145/1809939.1809959
Greenberg, S., Buxton, B.: Usability evaluation considered harmful (some of the time). In: Proc. CHI, pp. 111–120. ACM, New York (2008). doi:10.1145/1357054.1357074
Halper, N., Mellin, M., Herrmann, C.S., Linneweber, V., Strothotte, T.: Psychology and non-photorealistic rendering: the beginning of a beautiful relationship. In: Proc. Mensch & Computer, pp. 277–286. Teubner, Stuttgart (2003)
Halper, N., Mellin, M., Herrmann, C.S., Linneweber, V., Strothotte, T.: Towards an understanding of the psychology of non-photorealistic rendering. In: Proc. Workshop Computational Visualistics, Media Informatics and Virtual Communities, pp. 67–78. Deutscher Universitäts, Wiesbaden (2003)
Healey, C.G., Enns, J.T.: Perception and painting: a search for effective, engaging visualizations. IEEE Comput. Graph. Appl. 22(2), 10–15 (2002). doi:10.1109/38.988741
Healey, C.G., Tateosian, L., Enns, J.T., Remple, M.: Perceptually-based brush strokes for nonphotorealistic visualization. ACM Trans. Graph. 23(1), 64–96 (2004). doi:10.1145/966131.966135
Hertzmann, A.: A survey of stroke-based rendering. IEEE Comput. Graph. Appl. 23(4), 70–81 (2003). doi:10.1109/MCG.2003.1210867
Hertzmann, A.: Non-photorealistic rendering and the science of art. In: Proc. NPAR, pp. 147–157. ACM, New York (2010). doi:10.1145/1809939.1809957
Igarashi, T., Matsuoka, S., Tanaka, H.T.: A sketching interface for 3D freeform design. In: Proc. SIGGRAPH, pp. 409–416. ACM, New York (1999). doi:10.1145/311535.311602
Inglis, T.C., Kaplany, C.S.: Pixelating vector line art. In: Proc. NPAR, pp. 21–28. Eurographics Association, Goslar (2012). doi:10.2312/PE/NPAR/NPAR12/021-028
Isenberg, T., Freudenberg, B., Halper, N., Schlechtweg, S., Strothotte, T.: A developer’s guide to silhouette algorithms for polygonal models. IEEE Comput. Graph. Appl. 23(4), 28–37 (2003). doi:10.1109/MCG.2003.1210862
Isenberg, T., Neumann, P., Carpendale, S., Sousa, M.C., Jorge, J.A.: Non-photorealistic rendering in context: an observational study. In: Proc. NPAR, pp. 115–126. ACM, New York (2006). doi:10.1145/1124728.1124747
Jackson, C.D., Acevedo, D., Laidlaw, D.H., Drury, F., Vote, E., Keefe, D.: Designer-critiqued comparison of 2D vector visualization methods: a pilot study. In: ACM SIGGRAPH Sketches & Applications. ACM, New York (2003). doi:10.1145/965400.965505
Kaptein, M., Robertson, J.: Rethinking statistical analysis methods for CHI. In: Proc. CHI, pp. 1105–1114. ACM, New York (2012). doi:10.1145/2207676.2208557
Keefe, D.F., Karelitz, D.B., Vote, E.L., Laidlaw, D.H.: Artistic collaboration in designing VR visualizations. IEEE Comput. Graph. Appl. 25(2), 18–23 (2005). doi:10.1109/MCG.2005.34
Kerlinger, F.N., Lee, H.B.: Foundations of Behavioral Research, 4th edn. Wadsworth Publishing/Thomson Learning, London (2000)
Kim, S., Hagh-Shenas, H., Interrante, V.: Conveying shape with texture: an experimental investigation of the impact of texture type on shape categorization judgments. In: Proc. InfoVis, pp. 163–170. IEEE Comput. Soc., Los Alamitos (2003). doi:10.1109/INFVIS.2003.1249022
Kim, S., Hagh-Shenas, H., Interrante, V.: Conveying shape with texture: experimental investigation of texture’s effects on shape categorization judgments. IEEE Trans. Vis. Comput. Graph. 10(4), 471–483 (2004). doi:10.1109/TVCG.2004.5
Laidlaw, D.H.: Loose, artistic “textures” for visualization. IEEE Comput. Graph. Appl. 21(2), 6–9 (2001). doi:10.1109/38.909009
Laidlaw, D., Kirby, R., Jackson, C., Davidson, J., Miller, T., da Silva, M., Warren, W., Tarr, M.: Comparing 2D vector field visualization methods: a user study. IEEE Trans. Vis. Comput. Graph. 11(1), 59–70 (2005). doi:10.1109/TVCG.2005.4
Maciejewski, R., Isenberg, T., Andrews, W.M., Ebert, D.S., Sousa, M.C.: Aesthetics of hand-drawn vs. computer-generated stippling. In: Proc. CAe, pp. 53–56. Eurographics Association, Goslar (2007). doi:10.2312/COMPAESTH/COMPAESTH07/053-056
Maciejewski, R., Isenberg, T., Andrews, W.M., Ebert, D.S., Sousa, M.C., Chen, W.: Measuring stipple aesthetics in hand-drawn and computer-generated images. IEEE Comput. Graph. Appl. 28(2), 62–74 (2008). doi:10.1109/MCG.2008.35
Mandryk, R.L., Mould, D., Li, H.: Evaluation of emotional response to non-photorealistic images. In: Proc. NPAR, pp. 7–16. ACM, New York (2011). doi:10.1145/2024676.2024678
Martín, D., Arroyo, G., Luzón, M.V., Isenberg, T.: Example-based stippling using a scale-dependent grayscale process. In: Proc. NPAR, pp. 51–61. ACM, New York (2010). doi: 10.1145/1809939.1809946
Martín, D., Arroyo, G., Luzón, M.V., Isenberg, T.: Scale-dependent and example-based stippling. Comput. Graph. 35(1), 160–174 (2011). doi:10.1016/j.cag.2010.11.006
Mould, D., Mandryk, R.L., Li, H.: Emotional response and visual attention to non-photorealistic images. Comput. Graph. 36(5), 658–672 (2012). doi:10.1016/j.cag.2012.03.039
Rautek, P., Bruckner, S., Gröller, E., Viola, I.: Illustrative visualization: new technology or useless tautology? Comput. Graph. 42(3), 4:1–4:8 (2008). doi:10.1145/1408626.1408633
Saito, T., Takahashi, T.: Comprehensible rendering of 3-D shapes. Comput. Graph. 24(3), 197–206 (1990). doi:10.1145/97880.97901
Salesin, D.H.: Non-photorealistic animation & rendering: 7 grand challenges. Keynote talk at NPAR (2002)
Santella, A., DeCarlo, D.: Visual interest and NPR: an evaluation and manifesto. In: Proc. NPAR, pp. 71–78. ACM, New York (2004). doi:10.1145/987657.987669
Schmidt, R., Isenberg, T., Jepp, P., Singh, K., Wyvill, B.: Sketching, scaffolding, and inking: a visual history for interactive 3D modeling. In: Proc. NPAR, pp. 23–32. ACM, New York (2007). doi:10.1145/1274871.1274875
Schumann, J., Strothotte, T., Raab, A., Laser, S.: Assessing the effect of non-photorealistic rendered images in CAD. In: Proc. CHI, pp. 35–42. ACM, New York (1996). doi:10.1145/238386.238398
Seifi, H., DiPaola, S., Enns, J.: Exploring the effect of color palette in painterly rendered character sequences. In: Proc. CAe, pp. 89–97. Eurographics Association, Goslar (2012). doi:10.2312/COMPAESTH/COMPAESTH12/089-097
Strothotte, T., Preim, B., Raab, A., Schumann, J., Forsey, D.R.: How to render frames and influence people. Comput. Graph. Forum 13(3), 455–466 (1994). doi:10.1111/1467-8659.1330455
Tietjen, C., Isenberg, T., Preim, B.: Combining silhouettes, shading, and volume rendering for surgery education and planning. In: Proc. EuroVis, pp. 303–310. Eurographics Association, Goslar (2005). doi:10.2312/VisSym/EuroVis05/303-310
Winnemöller, H., Olsen, S.C., Gooch, B.: Real-time video abstraction. ACM Trans. Graph. 25(3), 1221–1226 (2006). doi:10.1145/1141911.1142018
Winnemöller, H., Feng, D., Gooch, B., Suzuki, S.: Using NPR to evaluate perceptual shape cues in dynamic environments. In: Proc. NPAR, pp. 85–92. ACM, New York (2007). doi:10.1145/1274871.1274885
Zhao, M., Zhu, S.C.: Sisley the abstract painter. In: Proc. NPAR, pp. 99–107. ACM, New York (2010). doi:10.1145/1809939.1809951
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Appendix: Data Resources
Appendix: Data Resources
Some of the datasets used/created in the mentioned studies are available online for further analysis and future studies. For example, the following datasets are available online at the moment of writing (of course, the URLs are always subject to change):
-
sparse line drawing comparison by Cole et al. [6, 7, 9];
→ captured registered drawings, models, etc.: http://gfx.cs.princeton.edu/proj/ld3d/
-
shape perception based on sparse line drawings by Cole et al. [6, 8];
→ gauge settings: http://gfx.cs.princeton.edu/proj/ld3d/
-
evaluation of the pixelization of line art by Inglis and Kaplan [28];
→ user study data: http://sites.google.com/site/tiffanycinglis/generating-pixel-art/generating-pixel-art---outlining
-
ethnographic study of illustrations by Isenberg et al. [30];
→ images: http://www.cs.rug.nl/~isenberg/VideosAndDemos/Isenberg2006NPR
-
shape perception in dynamic 3D environments by Winnemöller et al. [55];
→ 3D models: http://www.cs.northwestern.edu/~holger/Research/projects.htm
Rights and permissions
Copyright information
© 2013 Springer-Verlag London
About this chapter
Cite this chapter
Isenberg, T. (2013). Evaluating and Validating Non-photorealistic and Illustrative Rendering. In: Rosin, P., Collomosse, J. (eds) Image and Video-Based Artistic Stylisation. Computational Imaging and Vision, vol 42. Springer, London. https://doi.org/10.1007/978-1-4471-4519-6_15
Download citation
DOI: https://doi.org/10.1007/978-1-4471-4519-6_15
Published:
Publisher Name: Springer, London
Print ISBN: 978-1-4471-4518-9
Online ISBN: 978-1-4471-4519-6
eBook Packages: Computer ScienceComputer Science (R0)