Advertisement

Mechanical Novel: Crowdsourcing Complex Work Through Reflection and Revision

  • Joy Kim
  • Sarah Sterman
  • Allegra Argent Beal Cohen
  • Michael S. Bernstein
Chapter
Part of the Understanding Innovation book series (UNDINNO)

Abstract

Crowdsourcing systems accomplish large tasks with scale and speed by breaking work down into independent parts. However, many types of complex creative work, such as fiction writing, have remained out of reach for crowds because work is tightly interdependent: changing one part of a story may trigger changes to the overall plot and vice versa. Taking inspiration from how expert authors write, we propose a technique for achieving interdependent complex goals with crowds. With this technique, the crowd loops between reflection, to select a high-level goal, and revision, to decompose that goal into low-level, actionable tasks. We embody this approach in Mechanical Novel, a system that crowdsources short fiction stories on Amazon Mechanical Turk. In a field experiment, Mechanical Novel resulted in higher-quality stories than an iterative crowdsourcing workflow. Our findings suggest that orienting crowd work around high-level goals may enable workers to coordinate their effort to accomplish complex work.

Notes

Acknowledgements

We would like to thank Mechanical Turk workers and study participants for their time and valuable feedback. Thanks also to our colleagues who helped test early prototypes of Mechanical Novel. Special thanks to Kylie Jue for her help developing the Mechanical Novel system. This material is based upon work supported by the NSF under Grant No. DGE-114747 and Grant No. IIS-1351131 and by the Hasso Plattner Institute Design Thinking Research Program.

References

  1. Alexander, C., Ishikawa, S., & Silverstein, M. (1977). A pattern language (Vol. 2). Oxford: Oxford University Press.Google Scholar
  2. André, P., Kittur, A., & Dow, S. P. (2014). Crowd synthesis: Extracting categories and clusters from complex data. In Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work & Social Computing, CSCW ’14 (pp. 989–998). New York, NY: ACM. doi:10.1145/2531602.2531653. http://doi.acm.org/10.1145/2531602.2531653 Google Scholar
  3. Bayles, D., Orland, T., & Morey, A. (2012). Art & fear. Old Saybrook: Tantor Media, Incorporated.Google Scholar
  4. Bernstein, M. S., Little, G., Miller, R. C., Hartmann, B., Ackerman, M. S., Karger, D. R., Crowell, D., & Panovich, K. (2010). Soylent: A word processor with a crowd inside. In Proceedings of the 23Nd Annual ACM Symposium on User Interface Software and Technology, UIST ’10 (pp. 313–322). New York, NY: ACM. doi:10.1145/1866029.1866078. http://doi.acm.org/10.1145/1866029.1866078 CrossRefGoogle Scholar
  5. Bigham, J. P., Bernstein, M. S., & Adar, E. (2015). Human-computer interaction and collective intelligence. In T. W. Malone & M. S. Bernstein (Eds.), Collective intelligence handbook. Cambridge: MIT Press.Google Scholar
  6. Burroway, J. (2003). Imaginative writing: The elements of craft. New York: LongmanGoogle Scholar
  7. Chan, J., Dang, S., Kremer, P., Guo, L., & Dow, S. (2014). Ideagens: A social ideation system for guided crowd brainstorming. In Second AAAI Conference on Human Computation and Crowdsourcing.Google Scholar
  8. Chilton, L. B., Little, G., Edge, D., Weld, D. S., & Landay, J. A. (2013). Cascade: Crowdsourcing taxonomy creation. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’13 (pp. 1999–2008). New York, NY: ACM. doi:10.1145/2470654.2466265. http://doi.acm.org/10.1145/2470654.2466265
  9. d.school, S. (2016). Design method: I like, i wish, what if. http://dschool.stanford.edu/wp-content/themes/dschool/method-cards/i-like-i-wish-what-if.pdf
  10. Flower, L., & Hayes, J. R. (1981). A cognitive process theory of writing. College Composition and Communication, 32(4), 365–387.CrossRefGoogle Scholar
  11. Foldingstory. (2016). http://foldingstory.com/
  12. Hahn, N., Chang, J., Kim, J. E., & Kittur, A. (2016). The knowledge accelerator: Big picture thinking in small pieces. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, CHI ’16 (pp. 2258–2270). New York, NY: ACM. doi:10.1145/2858036.2858364. http://doi.acm.org/10.1145/2858036.2858364 Google Scholar
  13. Hill, B. M., & Monroy-Hernández, A. (2013). The cost of collaboration for code and art: Evidence from a remixing community. In Proceedings of the 2013 Conference on Computer Supported Cooperative Work, CSCW ’13 (pp. 1035–1046). New York, NY: ACM. doi:10.1145/2441776.2441893. http://doi.acm.org/10.1145/2441776.2441893 Google Scholar
  14. Jansson, D. G., & Smith, S. M. (1991). Design fixation. Design Studies, 12(1), 3–11.CrossRefGoogle Scholar
  15. Kim, H. C. E., & Eklundh, K. S. (2001). Reviewing practices in collaborative writing. Computer Supported Cooperative Work, 10(2), 247–259. doi:10.1023/A:1011229212323. http://dx.doi.org/10.1023/A:1011229212323 CrossRefGoogle Scholar
  16. Kim, J., Cheng, J., & Bernstein, M. S. (2014). Ensemble: Exploring complementary strengths of leaders and crowds in creative collaboration. In Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work & Social Computing, CSCW ’14 (pp. 745–755). New York, NY: ACM. doi:10.1145/2531602.2531638. http://doi.acm.org/10.1145/2531602.2531638 Google Scholar
  17. Kim, J., & Monroy-Hernández, A. (2016). Storia: Summarizing social media content based on narrative theory using crowdsourcing. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing, CSCW ’16 (pp. 1018–1027). New York: ACM. doi:10.1145/2818048.2820072. http://doi.acm.org/10.1145/2818048.2820072 Google Scholar
  18. Kittur, A., Smus, B., Khamkar, S., & Kraut, R. E. (2011). Crowdforge: Crowdsourcing complex work. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, UIST ’11 (pp. 43–52). New York, NY: ACM. doi:10.1145/2047196.2047202. http://doi.acm.org/10.1145/2047196.2047202 Google Scholar
  19. Koblin, A. (2010). The Johnny cash project. http://www.thejohnnycashproject.com/ Google Scholar
  20. Kulkarni, A., Can, M., & Hartmann, B. (2012). Collaboratively crowdsourcing workflows with turkomatic. In Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work, CSCW ’12 (pp. 1003–1012). New York, NY: ACM. doi:10.1145/2145204.2145354. http://doi.acm.org/10.1145/2145204.2145354 Google Scholar
  21. Lasecki, W., Miller, C., Sadilek, A., Abumoussa, A., Borrello, D., Kushalnagar, R., & Bigham, J. (2012). Real-time captioning by groups of non-experts. In Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology, UIST ’12 (pp. 23–34). New York, NY: ACM. doi:10.1145/2380116.2380122. http://doi.acm.org/10.1145/2380116.2380122 CrossRefGoogle Scholar
  22. Lasecki, W. S., Kim, J., Rafter, N., Sen, O., Bigham, J. P., & Bernstein, M. S. (2015). Apparition: Crowdsourced user interfaces that come to life as you sketch them. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, CHI ’15 (pp. 1925–1934). New York, NY: ACM. doi:10.1145/2702123.2702565. http://doi.acm.org/10.1145/2702123.2702565
  23. Lasecki, W. S., Wesley, R., Nichols, J., Kulkarni, A., Allen, J. F., & Bigham, J. P. (2013). Chorus: A crowd-powered conversational assistant. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology, UIST ’13 (pp. 151–162). New York, NY: ACM. doi:10.1145/2501988.2502057. http://doi.acm.org/10.1145/2501988.2502057 CrossRefGoogle Scholar
  24. Little, G., Chilton, L. B., Goldman, M., & Miller, R. C. (2010a). Exploring iterative and parallel human computation processes. In Proceedings of the ACM SIGKDD Workshop on Human Computation, HCOMP ’10 (pp. 68–76). New York, NY: ACM. doi:10.1145/1837885.1837907. http://doi.acm.org/10.1145/1837885.1837907
  25. Little, G., Chilton, L. B., Goldman, M., & Miller, R. C. (2010b). Turkit: Human computation algorithms on mechanical turk. In Proceedings of the 23nd Annual ACM Symposium on User Interface Software and Technology, UIST ’10 (pp. 57–66). New York, NY: ACM. doi:10.1145/1866029.1866040. http://doi.acm.org/10.1145/1866029.1866040
  26. Luther, K., Fiesler, C., & Bruckman, A. (2013). Redistributing leadership in online creative collaboration. In Proceedings of the 2013 Conference on Computer Supported Cooperative Work, CSCW ’13 (pp. 1007–1022). New York, NY: ACM. doi:10.1145/2441776.2441891. http://doi.acm.org/10.1145/2441776.2441891 Google Scholar
  27. Mason, B., & Thomas, S. (2008). A million penguins research report. Leicester: Institute of Creative Technologies, De Montfort University.Google Scholar
  28. Nebeling, M., To, A., Guo, A., de Freitas, A. A., Teevan, J., Dow, S. P., & Bigham, J. P. (2016). Wearwrite: Crowd-assisted writing from smart watches. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, CHI ’16 (pp. 3834–3846). New York, NY: ACM. doi:10.1145/2858036.2858169. http://doi.acm.org/10.1145/2858036.2858169 Google Scholar
  29. Noël, S., & Robert, J. M. (2004). Empirical study on collaborative writing: What do co-authors do, use, and like? Computer Supported Cooperative Work, 13(1), 63–89. doi:10.1023/B:COSU.0000014876.96003.be. http://dx.doi.org/10.1023/B:COSU.0000014876.96003.be CrossRefGoogle Scholar
  30. Pecher, D., & Zwaan, R. A. (2005). Grounding cognition: The role of perception and action in memory, language, and thinking. Cambridge: Cambridge University Press.CrossRefGoogle Scholar
  31. Retelny, D., Robaszkiewicz, S., To, A., Lasecki, W. S., Patel, J., Rahmati, N., Doshi, T., Valentine, M., & Bernstein, M. S. (2014). Expert crowdsourcing with flash teams. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology, UIST ’14 (pp. 75–85). New York, NY: ACM. doi:10.1145/2642918.2647409. http://doi.acm.org/10.1145/2642918.2647409 Google Scholar
  32. Schön, D. A. (1983). The reflective practitioner. London: Temple Smith.Google Scholar
  33. Sharples, M. (1999). An account of writing as creative design. Abingdon: Psychology Press.Google Scholar
  34. Simpson, R., Page, K. R., & De Roure, D. (2014). Zooniverse: Observing the world’s largest citizen science platform. In Proceedings of the 23rd International Conference on World Wide Web, WWW ’14 Companion (pp. 1049–1054). New York, NY: ACM. doi:10.1145/2567948.2579215. http://doi.acm.org/10.1145/2567948.2579215 CrossRefGoogle Scholar
  35. Teevan, J., Iqbal, S. T., & von Veh, C. (2016). Supporting collaborative writing with microtasks. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, CHI ’16 (pp. 2657–2668). New York, NY: ACM. doi:10.1145/2858036.2858108. http://doi.acm.org/10.1145/2858036.2858108 Google Scholar
  36. Thompson, J. D. (1967). Organizations in action: Social science bases of administrative theory. Piscataway: Transaction Publishers.Google Scholar
  37. Verroios, V., & Bernstein, M. S. (2014). Context trees: Crowdsourcing global understanding from local views. In Proceedings of the Second AAAI Conference on Human Computation and Crowdsourcing. http://ilpubs.stanford.edu:8090/1105/
  38. WeAreDynamo: Guidelines for academic requesters. http://wiki.wearedynamo.org/index.php/Guidelines_for_Academic_Requesters Accessed May 15, 2014.
  39. Xu, A., Huang, S. W., & Bailey, B. (2014). Voyant: Generating structured feedback on visual designs using a crowd of non-experts. In Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work & Social Computing, CSCW ’14 (pp. 1433–1444). New York, NY: ACM. doi:10.1145/2531602.2531604. http://doi.acm.org/10.1145/2531602.2531604 Google Scholar
  40. Yu, L., & Nickerson, J. V. (2011). Cooks or cobblers?: Crowd creativity through combination. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’11 (pp. 1393–1402). New York, NY: ACM. doi:10.1145/1978942.1979147. http://doi.acm.org/10.1145/1978942.1979147 Google Scholar
  41. Zhang, H., Law, E., Miller, R., Gajos, K., Parkes, D., & Horvitz, E. (2012). Human computation tasks with global constraints. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’12 (pp. 217–226). New York, NY: ACM. doi:10.1145/2207676.2207708. http://doi.acm.org/10.1145/2207676.2207708 Google Scholar

Copyright information

© Springer International Publishing AG 2018

Authors and Affiliations

  • Joy Kim
    • 1
  • Sarah Sterman
    • 1
  • Allegra Argent Beal Cohen
    • 2
  • Michael S. Bernstein
    • 1
  1. 1.Computer ScienceStanford UniversityStanfordUSA
  2. 2.Symbolic SystemsStanford UniversityStanfordUSA

Personalised recommendations