Skip to main content
SpringerLink
Log in

Crowd Research: Open and Scalable University Laboratories

  • Chapter
  • First Online:
Design Thinking Research

Part of the book series: Understanding Innovation ((UNDINNO))

Abstract

Research experiences today are limited to a privileged few at select universities. Providing open access to research experiences would enable global upward mobility and increased diversity in the scientific workforce. How can we coordinate a crowd of diverse volunteers on open-ended research? How could a PI have enough visibility into each person’s contributions to recommend them for further study? We present Crowd Research, a crowdsourcing technique that coordinates open-ended research through an iterative cycle of open contribution, synchronous collaboration, and peer assessment. To aid upward mobility and recognize contributions in publications, we introduce a decentralized credit system: participants allocate credits to each other, which a graph centrality algorithm translates into a collectively-created author order. Over 1500 people from 62 countries have participated, 74% from institutions with low access to research. Over 2 years and three projects, this crowd has produced articles at top-tier Computer Science venues, and participants have gone on to leading graduate programs.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 79.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 99.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 139.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  • Baeza-Yates, R. A., Castillo, C., LĂłpez, V., Shubik, M., Hopcroft, J., & Sheldon, D. (2007). Pagerank increase under different collusion topologies. In AIRWeb (Vol. 5, pp. 68–81). New York: Springer.

    Google Scholar 

  • Bianchini, J. A. (2011). Expanding underrepresented minority participation: America’s science and technology talent at the crossroads. Washington: National Academies.

    Google Scholar 

  • Bonney, R., Ballard, H., Jordan, R., McCallie, E., Phillips, T., Shirk, J., & Wilderman, C. C. (2009). Public participation in scientific research: Defining the field and assessing its potential for informal science education.

    Google Scholar 

  • Bowen, W. G., & Bok, D. (2016). The shape of the river: Long-term consequences of considering race in college and university admissions. Princeton: Princeton University Press.

    Google Scholar 

  • Budhathoki, N. (2016). Who are the mappers and why do they map in OpenStreetMap. https://www.youtube.com/watch?v=LvakiUOsDrM

  • Chen, Y., Oney, S., & Lasecki, W. S. (2016). Towards providing on-demand expert support for software developers. In Proceedings of the 2016 CHI conference on human factors in computing systems (pp. 3192–3203). New York: ACM.

    Chapter  Google Scholar 

  • CIA. (2016). The world factbook. https://www.cia.gov/library/publications/the-world-factbook/

  • Cooper, S., Khatib, F., Treuille, A., Barbero, J., Lee, J., Beenen, M., Leaver-Fay, A., Baker, D., Popovic, Z., & Players, F. (2010). Predicting protein structures with a multiplayer online game. Nature, 466(7307), 756–760.

    Article  Google Scholar 

  • Cox, J., Eun Young, O., Simmons, B., Lintott, C., Masters, K., Greenhill, A., Graham, G., & Holmes, K. (2015). Defining and measuring success in online citizen science: A case study of Zooniverse projects. Computing in Science & Engineering, 17(4), 28–41.

    Article  Google Scholar 

  • Cramton, C. D., & Hinds, P. J. (2004). Subgroup dynamics in internationally distributed teams: Ethnocentrism or cross-national learning? Research in Organizational Behavior, 26, 231–263.

    Article  Google Scholar 

  • Cranshaw, J., & Kittur, A. (2011). The polymath project: Lessons from a successful online collaboration in mathematics. In Proceedings of the SIGCHI conference on human factors in computing systems (pp. 1865–1874). New York: ACM.

    Chapter  Google Scholar 

  • Cronje, R., Rohlinger, S., Crall, A., & Newman, G. (2011). Does participation in citizen science improve scientific literacy? A study to compare assessment methods. Applied Environmental Education & Communication, 10(3), 135–145.

    Article  Google Scholar 

  • Dillahunt, T. R., Chen, B., & Teasley, S. (2014). Model thinking: Demographics and performance of MOOC students unable to afford a formal education. In Proceedings of the first ACM conference on learning@ scale conference (pp. 145–146). New York: ACM.

    Chapter  Google Scholar 

  • Dillahunt, T. R., Ng, S., Fiesta, M., & Wang, Z. (2016). Do massive open online course platforms support employability? In Proceedings of the 19th ACM conference on computer-supported cooperative work & social computing (pp. 232–243). New York: ACM.

    Google Scholar 

  • Ding, Y., Yan, E., Frazho, A., & Caverlee, J. (2009). PageRank for ranking authors in co-citation networks. Journal of the American Society for Information Science and Technology, 60(11), 2229–2243.

    Article  Google Scholar 

  • Dow, S., Kulkarni, A., Klemmer, S., & Hartmann, B. (2012). Shepherding the crowd yields better work. In Proceedings of the ACM 2012 conference on computer supported cooperative work (p. 1013). New York: ACM.

    Chapter  Google Scholar 

  • Eric Landrum, R., & Nelsen, L. R. (2002). The undergraduate research assistantship: An analysis of the benefits. Teaching of Psychology, 29(1), 15–19.

    Article  Google Scholar 

  • Gaikwad, S. S., Morina, D., Ginzberg, A., Mullings, C., Goyal, S., Gamage, D., Diemert, C., Burton, M., Zhou, S., Whiting, M., Ziulkoski, K., Ballav, A., Gilbee, A., Niranga, S. S., Sehgal, V., Lin, J., Kristianto, L., Regino, J., Chhibber, N., Majeti, D., Sharma, S., Mananova, K., Dhakal, D., Dai, W., Purynova, V., Sandeep, S., Chandrakanthan, V., Sarma, T., Matin, S., Nassar, A., Nistala, R., Stolzoff, A., Milland, K., Mathur, V., Vaish, R., & Bernstein, M. S. (2016). Boomerang: Rebounding the consequences of reputation feedback on crowdsourcing platforms. In Proceedings of the 29th annual ACM symposium on user interface software and technology (UIST ’16). New York: ACM.

    Google Scholar 

  • Gaikwad, S. S., Morina, D., Nistala, R., Agarwal, M., Cossette, A., Bhanu, R., Savage, S., Narwal, V., Rajpal, K., Regino, J., Mithal, A., Ginzberg, A., Nath, A., Ziulkoski, K. R., Cossette, T., Gamage, D., Richmond-Fuller, A., Suzuki, R., HerrejĂłn, J., Le, K., Flores-Saviaga, C., Thilakarathne, H., Gupta, K., Dai, W., Sastry, A., Goyal, S., Rajapakshe, T., Abolhassani, N., Xie, A., Reyes, A., Ingle, S., Jaramillo, V., Godinez, M., Angel, W., Toxtli, C., Flores, J., Gupta, A., Sethia, V., Padilla, D., Milland, K., Setyadi, K., Wajirasena, N., Batagoda, M., Cruz, R., Damon, J., Nekkanti, D., Sarma, T., Saleh, M., Gongora-Svartzman, G., Bateni, S., Barrera, G. T., Pena, A., Compton, R., Aariff, D., Palacios, L., Ritter, M. P., Nisha, K. K., Kay, A., Uhrmeister, J., Nistala, S., Esfahani, M., Bakiu, E., Diemert, C., Matsumoto, L., Singh, M., Patel, K., Krishna, R., Kovacs, G., Vaish, R., & Bernstein, M. (2015). Daemo: A self-governed crowdsourcing marketplace. In Adjunct proceedings of the 28th annual ACM symposium on user interface software & technology (pp. 101–102). New York: ACM.

    Google Scholar 

  • Gaikwad, S. S., Whiting, M. E., Gamage, D., Mullings, C. A., Majeti, D., Goyal, S., Gilbee, A., Chhibber, N., Ginzberg, A., Richmond-Fuller, A., Matin, S., Sehgal, V., Sarma, T. S., Nasser, A., Ballav, A., Regino, J., Zhou, S., Mananova, K., Srinivas, P., Ziulkoski, K., Dhakal, D., Stolzoff, A., Niranga, S. S., Salih, M. H., Sinha, A., Vaish, R., & Bernstein, M. S. (2017). The daemo crowdsourcing marketplace. In Companion of the 2017 ACM conference on computer supported cooperative work and social computing (pp. 1–4). New York: ACM.

    Google Scholar 

  • Gerstner, C. R., & Day, D. V. (1997). Meta-analytic review of leader-member exchange theory: Correlates and construct issues. Journal of Applied Psychology, 82(6), 827–844.

    Article  Google Scholar 

  • Gilbert, E. (2013). Widespread underprovision on Reddit. In Proceedings of the 2013 conference on computer supported cooperative work (pp. 803–808). New York: ACM.

    Chapter  Google Scholar 

  • Gowers, T., & Nielsen, M. (2009). Massively collaborative mathematics. Nature, 461(7266), 879–881.

    Article  Google Scholar 

  • Gurin, P., Dey, E., Hurtado, S., & Gurin, G. (2002). Diversity and higher education: Theory and impact on educational outcomes. Harvard Educational Review, 72(3), 330–367.

    Article  Google Scholar 

  • Haizi, Y., Deka, B., Talton, J. O., & Kumar, R. (2016). Accounting for taste: Ranking curators and content in social networks. In Proceedings of the 2016 CHI conference on human factors in computing systems (pp. 2383–2389). New York: ACM.

    Google Scholar 

  • Hathaway, R. S., Nagda, B. A., & Gregerman, S. R. (2002). The relationship of undergraduate research participation to graduate and professional education pursuit: An empirical study. Journal of College Student Development, 43(5), 614.

    Google Scholar 

  • Hinds, P. J., & Bailey, D. E. (2003). Out of sight, out of sync: Understanding conflict in distributed teams. Organization Science, 14(6), 615–632.

    Article  Google Scholar 

  • Hinds, P. J., & Mortensen, M. (2005). Understanding conflict in geographically distributed teams: The moderating effects of shared identity, shared context, and spontaneous communication. Organization Science, 16(3), 290–307.

    Article  Google Scholar 

  • Hong, L., & Page, S. E. (2004). Groups of diverse problem solvers can outperform groups of high-ability problem solvers. Proceedings of the National Academy of Sciences, 101(46), 16385–16389.

    Article  Google Scholar 

  • Hunter, A.-B., Laursen, S. L., & Seymour, E. (2007). Becoming a scientist: The role of undergraduate research in students’ cognitive, personal, and professional development. Science Education, 91(1), 36–74.

    Article  Google Scholar 

  • Irani, L. C., & Silberman, M. (2013). Turkopticon: Interrupting worker invisibility in amazon mechanical turk. In Proceedings of the SIGCHI conference on human factors in computing systems (pp. 611–620). New York: ACM.

    Chapter  Google Scholar 

  • Jordan Raddick, M., Bracey, G., Gay, P. L., Lintott, C. J., Murray, P., Schawinski, K., Szalay, A. S., & Vandenberg, J. (2009). Galaxy zoo: Exploring the motivations of citizen science volunteers. arXiv preprint arXiv:0909.2925.

    Google Scholar 

  • Kevin Eagan, M., Hurtado, S., Chang, M. J., Garcia, G. A., Herrera, F. A., Garibay, J. C., & Garibay, J. C. (2013). Making a difference in science education: The impact of undergraduate research programs. American Educational Research Journal, 50(4), 683–713.

    Article  Google Scholar 

  • Khatib, F., DiMaio, F., Cooper, S., Kazmierczyk, M., Gilski, M., Krzywda, S., Zabranska, H., Pichova, I., Thompson, J., Popovic, Z., Jaskolski, M., & Baker, D. (2011). Crystal structure of a monomeric retroviral protease solved by protein folding game players. Nature Structural & Molecular Biology, 18(10), 1175–1177.

    Article  Google Scholar 

  • Kiesler, S., Kraut, R., Resnick, P., & Kittur, A. (2012). Regulating behavior in online communities. In Building successful online communities: Evidence-based social design. Cambridge: MIT.

    Google Scholar 

  • Kittur, A., Nickerson, J. V., Bernstein, M., Gerber, E., Shaw, A., Zimmerman, J., Lease, M., & Horton, J. (2013). The future of crowd work. In Proceedings of the 2013 conference on computer supported cooperative work (p. 1301). New York: ACM.

    Chapter  Google Scholar 

  • Kittur, A., Smus, B., Khamkar, S., & Kraut, R. E. (2011). Crowdforge: Crowdsourcing complex work. In Proceedings of the 24th annual ACM symposium on user interface software and technology (pp. 43–52). New York: ACM.

    Chapter  Google Scholar 

  • Kizilcec, R. F., & Halawa, S. (2015). Attrition and achievement gaps in online learning. In Proceedings of the second ACM conference on learning@ scale (pp. 57–66). New York: ACM.

    Chapter  Google Scholar 

  • Kleinberg, J. M. (1999). Authoritative sources in a hyperlinked environment. Journal of the ACM (JACM), 46(5), 604–632.

    Article  Google Scholar 

  • Kulkarni, C. E., Bernstein, M. S., & Klemmer, S. R. (2015). PeerStudio: Rapid peer feedback emphasizes revision and improves performance. In Proceedings of the second (2015) ACM conference on learning@ scale (pp. 75–84). New York: ACM.

    Chapter  Google Scholar 

  • Kulkarni, C., Wei, K. P., Le, H., Chia, D., Papadopoulos, K., Cheng, J., Koller, D., & Klemmer, S. R. (2013). Peer and self assessment in massive online classes. ACM Transactions on Computer-Human Interaction (TOCHI), 20(6), 33.

    Article  Google Scholar 

  • Lakhani, K. R., Lifshitz-Assaf, H., & Tushman, M. (2013). Open innovation and organizational boundaries: Task decomposition, knowledge distribution and the locus of innovation. In Handbook of economic organization: Integrating economic and organizational theory (pp. 355–382). Cheltenham: Edward Elgar.

    Google Scholar 

  • Land, K., Slosar, A., Lintott, C., Andreescu, D., Bamford, S., Murray, P., Robert, N., Jordan Raddick, M., Schawinski, K., Szalay, A., Thomas, D., & Vandenberg, J. (2008). Galaxy zoo: The large-scale spin statistics of spiral galaxies in the sloan digital sky survey. Monthly Notices of the Royal Astronomical Society, 388(4), 1686–1692.

    Article  Google Scholar 

  • Lashinsky, A. (2012). Inside apple: How America’s most admired–and secretive–company really works. London: John Murray.

    Google Scholar 

  • LaToza, T. D., Ben Towne, W., Adriano, C. M., & Van Der Hoek, A. (2014). Microtask programming: Building software with a crowd. In Proceedings of the 27th annual ACM symposium on user interface software and technology (pp. 43–54). New York: ACM.

    Chapter  Google Scholar 

  • Law, E., Wiggins, A., Gray, M. L., & Williams, A. (2017). Crowdsourcing as a tool for research: Implications of uncertainty. In Proceedings of the 20th ACM conference on computer-supported cooperative work & social computing (CSCW ’17). New York: ACM.

    Google Scholar 

  • Lee, J., Kladwang, W., Lee, M., Cantu, D., Azizyan, M., Kim, H., Limpaecher, A., Gaikwad, S., Yoon, S., Treuille, A., et al. (2014). RNA design rules from a massive open laboratory. Proceedings of the National Academy of Sciences, 111(6), 2122–2127.

    Article  Google Scholar 

  • Li, J. (2003). US and Chinese cultural beliefs about learning. Journal of Educational Psychology, 95(2), 258.

    Article  Google Scholar 

  • Lixiu, Y., Kittur, A., & Kraut, R. E. (2014). Distributed analogical idea generation: Inventing with crowds. In Proceedings of the 32nd annual ACM conference on human factors in computing systems (pp. 1245–1254). New York: ACM.

    Google Scholar 

  • Lopatto, D. (2004). Survey of undergraduate research experiences (SURE): First findings. Cell Biology Education, 3(4), 270–277.

    Article  Google Scholar 

  • Mao, A., Mason, W., Suri, S., Watts, D. J., & Malone, T. W. (2016). An experimental study of team size and performance on a complex task. PLoS One, 11(4), e0153048.

    Article  Google Scholar 

  • Martin, D., Hanrahan, B. V., O’Neill, J., & Gupta, N. (2014). Being a turker. In Proceedings of the 17th ACM conference on computer supported cooperative work & social computing (pp. 224–235). New York: ACM.

    Google Scholar 

  • Miller, R. C., Zhang, H., Gilbert, E., & Gerber, E. (2014). Pair research: Matching people for collaboration, learning, and productivity. In Proceedings of the 17th ACM conference on computer supported cooperative work & social computing (pp. 1043–1048). New York: ACM.

    Google Scholar 

  • Mysore, A. S., Yaligar, V. S., Ibarra, I. A., Simoiu, C., Goel, S., Arvind, R., Sumanth, C., Srikantan, A., Bhargav, H. S., Pahadia, M., Dobha, T., Ahmed, A., Shankar, M., Agarwal, H., Agarwal, R., Anirudh-Kondaveeti, S., Arun-Gokhale, S., Attri, A., Chandra, A., Chilukur, Y., Dharmaji, S., Garg, D., Gupta, N., Gupta, P., Jacob, G. M., Jain, S., Joshi, S., Khajuria, T., Khillan, S., Konam, S., Kumar-Kolla, P., Loomba, S., Madan, R., Maharaja, A., Mathur, V., Munshi, B., Nawazish, M., Neehar-Kurukunda, V., Nirmal-Gavarraju, V., Parashar, S., Parikh, H., Paritala, A., Patil, A., Phatak, R., Pradhan, M., Ravichander, A., Sangeeth, K., Sankaranarayanan, S., Sehgal, V., Sheshan, A., Shibiraj, S., Singh, A., Singh, A., Sinha, P., Soni, P., Thomas, B., Varma-Dattada, K., Venkataraman, S., Verma, P., & Yelurwar, I. (2015). Investigating the “wisdom of crowds” at scale. In Adjunct proceedings of the 28th annual ACM symposium on user interface software & technology (UIST ’15 adjunct) (pp. 75–76). New York: ACM.

    Google Scholar 

  • Nebeling, M., To, A., Guo, A., de Freitas, A. A., Teevan, J., Dow, S. P., & Bigham, J. P. (2016). WearWrite: Crowd-assisted writing from smartwatches. In Proceedings of the 2016 CHI conference on human factors in computing systems (pp. 3834–3846). New York: ACM.

    Chapter  Google Scholar 

  • Oliveira, N., Jun, E., & Reinecke, K. (2017). Citizen science opportunities in volunteer-based online experiments. In Proceedings of the 2017 CHI conference on human factors in computing systems (pp. 6800–6812). New York: ACM.

    Chapter  Google Scholar 

  • Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349, 6251.

    Article  Google Scholar 

  • Page, S. E. (2008). The difference: How the power of diversity creates better groups, firms, schools, and societies. Princeton: Princeton University Press.

    Book  Google Scholar 

  • Page, L., Brin, S., Motwani, R., & Winograd, T. (1999). The PageRank citation ranking: Bringing order to the web. Stanford: Stanford InfoLab.

    Google Scholar 

  • Pandey, V., Amir, A., Debelius, J., Hyde, E. R., Kosciolek, T., Knight, R., & Klemmer, S. (2017). Gut instinct: Creating scientific theories with online learners. In Proceedings of the 2017 CHI conference on human factors in computing systems (pp. 6825–6836). New York: ACM.

    Chapter  Google Scholar 

  • Patterson, D. A. (1994). How to have a bad career in research/academia. In Keynote, 1994 USENIX symposium on operating system design and implementation.

    Google Scholar 

  • Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016). Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 conference on empirical methods in natural language processing.

    Google Scholar 

  • Reinecke, K., & Gajos, K. Z. (2015). LabintheWild: Conducting large-scale online experiments with uncompensated samples. In Proceedings of the 18th ACM conference on computer supported cooperative work & social computing (pp. 1364–1378). New York: ACM.

    Google Scholar 

  • Retelny, D., Robaszkiewicz, S., To, A., Lasecki, W. S., Patel, J., Rahmati, N., Doshi, T., Valentine, M., & Bernstein, M. S. (2014). Expert crowdsourcing with flash teams. In Proceedings of the 27th annual ACM symposium on user interface software and technology (pp. 75–85). New York: ACM.

    Chapter  Google Scholar 

  • Russakovsky, O., Li, L.-J., & Li, F.-F. (2015). Best of both worlds: Human-machine collaboration for object annotation. In IEEE conference on computer vision and pattern recognition (CVPR) (pp. 2121–2131). Boston: IEEE.

    Google Scholar 

  • Russell, S. H., Hancock, M. P., & McCullough, J. (2007). Benefits of undergraduate research experiences. Science, 316(5824), 548–549.

    Article  Google Scholar 

  • Sarsons, H. (2015). Gender differences in recognition for group work. Working Paper, Harvard University.

    Google Scholar 

  • Shen, H.-W., & Barabási, A.-L. (2014). Collective credit allocation in science. Proceedings of the National Academy of Sciences, 111(34), 12325–12330.

    Article  Google Scholar 

  • Silvertown, J. (2009). A new dawn for citizen science. Trends in Ecology & Evolution, 24(9), 467–471.

    Article  Google Scholar 

  • Stahl, G. K., Maznevski, M. L., Voigt, A., & Jonsen, K. (2010). Unraveling the effects of cultural diversity in teams: A meta-analysis of research on multicultural work groups. Journal of International Business Studies, 41(4), 690–709.

    Article  Google Scholar 

  • Stevens, M., Vitos, M., Altenbuchner, J., Conquest, G., Lewis, J., & Haklay, M. (2014). Taking participatory citizen science to extremes. IEEE Pervasive Computing, 13(2), 20–29.

    Article  Google Scholar 

  • Sullivan, B. L., Wood, C. L., Iliff, M. J., Bonney, R. E., Fink, D., & Kelling, S. (2009). eBird: A citizen-based bird observation network in the biological sciences. Biological Conservation, 142(10), 2282–2292.

    Article  Google Scholar 

  • Sun, Y., Yintao, Y., & Han, J. (2009). Ranking-based clustering of heterogeneous information networks with star network schema. In Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining (pp. 797–806). New York: ACM.

    Chapter  Google Scholar 

  • Suzuki, R., Salehi, N., Lam, M. S., Marroquin, J. C., & Bernstein, M. S. (2016). Atelier: Repurposing expert crowdsourcing tasks as micro-internships. In Proceedings of the 2016 CHI conference on human factors in computing systems (pp. 2645–2656). New York: ACM.

    Chapter  Google Scholar 

  • Teevan, J., Iqbal, S. T., & von Veh, C. (2016). Supporting collaborative writing with microtasks. In Proceedings of the 2016 CHI conference on human factors in computing systems (pp. 2657–2668). New York: ACM.

    Chapter  Google Scholar 

  • The Times Higher Education World University Rankings. (2017). https://www.timeshighereducation.com/world-university-rankings

  • Toegel, G., & Conger, J. A. (2003). 360-degree assessment: Time for reinvention. Academy of Management Learning & Education, 2(3), 297–311.

    Article  Google Scholar 

  • Tomlinson, B., Ross, J., Andre, P., Baumer, E., Patterson, D., Corneli, J., Mahaux, M., Nobarany, S., Lazzari, M., Penzenstadler, B., et al. (2012). Massively distributed authorship of academic papers. In Extended abstracts of the 2012 ACM annual conference on human factors in computing systems (pp. 11–20). New York: ACM.

    Google Scholar 

  • Tsai, J. L. (2007). Ideal affect: Cultural causes and behavioral consequences. Perspectives on Psychological Science, 2(3), 242–259.

    Article  Google Scholar 

  • Valentine, M. A., Retelny, D., To, A., Rahmati, N., Doshi, T., & Bernstein, M. S. (2017). Flash organizations: Crowdsourcing complex work by structuring crowds as organizations. In Proceedings of the 2017 CHI conference on human factors in computing systems (pp. 3523–3537). New York: ACM.

    Chapter  Google Scholar 

  • Veit, A., Wilber, M. J., Vaish, R., Belongie, S. J., Davis, J., Anand, V., Aviral, A., Chakrabarty, P., Chandak, Y., Chaturvedi, S., Devaraj, C., Dhall, A., Dwivedi, U., Gupte, S., Sridhar, S. N., Paga, K., Pahuja, A., Raisinghani, A., Sharma, A., Sharma, S., Sinha, D., Thakkar, N., Bala Vignesh, K., Verma, U., Abhishek, K., Agrawal, A., Aishwarya, A., Bhattacharjee, A., Dhanasekar, S., Gullapalli, V. K., Gupta, S., Chandana, G., Jain, K., Kapur, S., Kasula, M., Kumar, S., Kundaliya, P., Mathur, U., Mishra, A., Mudgal, A., Nadimpalli, A., Nihit, M. S., Periwal, A., Sagar, A., Shah, A., Sharma, V., Sharma, Y., Siddiqui, F., Singh, V., Abhinav, S., Tambwekar, P., Taskin, R., Tripathi, A., & Yadav, A. D. (2015). On optimizing human-machine task assignments. HCOMP 2015 extended abstracts.

    Google Scholar 

  • Whiting, M. E., Gamage, D., Snehalkumar (Neil) S Gaikwad, Gilbee, A., Goyal, S., Ballav, A., Majeti, D., Chhibber, N., Richmond-Fuller, A., Vargus, F., Sarma, T. S., Chandrakanthan, V., Moura, T., Salih, M. H., Kalejaiye, G. B. T., Ginzberg, A., Mullings, C. A., Dayan, Y., Milland, K., Orefice, H., Regino, J., Parsi, S., Mainali, K., Sehgal, V., Matin, S., Sinha, A., Vaish, R., & Bernstein, M. S. (2017). Crowd guilds: Worker-led reputation and feedback on crowdsourcing platforms. In Proceedings of the 2017 ACM conference on computer supported cooperative work and social computing (CSCW ’17) (pp. 1902–1913). New York: ACM.

    Google Scholar 

  • Zhang, H., Easterday, M. W., Gerber, E. M., Lewis, D. R., & Maliakal, L. (2017). Agile research studios. In Proceedings of the 2017 ACM conference on computer supported cooperative work and social computing – CSCW ’17 (pp. 220–232). New York: ACM.

    Google Scholar 

  • Zydney, A. L., Bennett, J. S., Shahid, A., & Bauer, K. W. (2002). Impact of undergraduate research experience in engineering. Journal of Engineering Education, 91, 151–157.

    Article  Google Scholar 

Download references

Acknowledgements

We thank over 1500 members of the Stanford Crowd Research Collective community for their contributions. This work was supported by Office of Naval Research awards N00014-16-1-2894 and N00014-15-1-2711, Institute for Scalable Scientific Data Management at UCSC and Los Alamos National Laboratory, Toyota, and the Hasso-Plattner Design Thinking Research Program.

Author information

Authors and Affiliations

  1. Stanford University, Stanford, CA, USA

    Rajan Vaish, Geza Kovacs, Ranjay Krishna, Imanol Arrieta Ibarra, Camelia Simoiu, Sharad C. Goel & Michael S. Bernstein

  2. MIT Media Lab, Cambridge, MA, USA

    Snehalkumar (Neil) S. Gaikwad

  3. Cornell Tech, New York, NY, USA

    Andreas Veit, Michael Wilber & Serge Belongie

  4. UC Santa Cruz, Santa Cruz, CA, USA

    James Davis

Authors
  1. Rajan Vaish
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Snehalkumar (Neil) S. Gaikwad
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Geza Kovacs
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Andreas Veit
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Ranjay Krishna
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Imanol Arrieta Ibarra
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Camelia Simoiu
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Michael Wilber
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Serge Belongie
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Sharad C. Goel
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. James Davis
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Michael S. Bernstein
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Rajan Vaish .

Editor information

Editors and Affiliations

  1. Hasso Plattner Institute for Digital Engineering, University of Potsdam, Potsdam, Germany

    Christoph Meinel

  2. Stanford University, Stanford, CA, USA

    Larry Leifer

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Vaish, R. et al. (2019). Crowd Research: Open and Scalable University Laboratories. In: Meinel, C., Leifer, L. (eds) Design Thinking Research. Understanding Innovation. Springer, Cham. https://doi.org/10.1007/978-3-319-97082-0_7

Download citation

Publish with us

Policies and ethics

Search

Navigation