Decision Making in Enterprise Crowdsourcing Services

  • Maja Vukovic
  • Rajarshi Das
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8274)


Enterprises are increasingly employing crowdsourcing to engage employees and public as part of their business processes, given a promising, low cost, access to scalable workforce online. Common examples include harnessing of crowd expertise for enterprise knowledge discovery, software development, product support and innovation. Crowdsourcing tasks vary in their complexity, required level of business support and investment, and most importantly the quality of outcome. As such, not every step in a business process can successfully lend itself to crowdsourcing. In this paper, we present a decision-making and execution service, called CrowdArb, operating on crowdsourcing tasks in the large global enterprise. The system employs decision theoretic methodology to assess whether to crowdsource or not a selected step of the knowledge discovery process. The system addresses the challenges of trade-off between the quality and time of the crowdsourcing responses, as well as the trade-off between the cost of crowdsourcing experts and time required to complete the entire campaign. We present evaluation results from simulations of CrowdArb in enterprise crowdsourcing campaign that engaged over 560 client representatives to obtain actionable insights. We discuss how proposed solution addresses the opportunity to close the gap of semi-automated task coordination in crowdsourcing environments.


Enterprise Organizational Services Crowdsourcing 


  1. 1.
    Vukovic, M.: Crowdsourcing for Enterprises. In: SERVICES, pp. 686–692 (2009)Google Scholar
  2. 2.
    Bayus, B.L.: Crowdsourcing New Product Ideas over Time: An Analysis of the Dell IdeaStorm Community. Management Science 59, 226–244 (2013) (published online before print November 5, 2012), doi:10.1287/mnsc.1120.1599Google Scholar
  3. 3.
    Brabham, D.C.: Moving the crowd at Threadless: Motivations for participation in a crowdsourcing application. Information, Communication & Society 13(8), 1122–1145 (2010)CrossRefGoogle Scholar
  4. 4.
    Lakhani, K., Garbin, D., Lonstein, E.: TopCoder (A), Developing Software through Crowdsourcing. Harvard Business School Case 610-032 (2009)Google Scholar
  5. 5.
    Vukovic, M., Natarajan, A.: Collective Intelligence for Enhanced Quality Management of IT Services. In: Liu, C., Ludwig, H., Toumani, F., Yu, Q. (eds.) Service Oriented Computing. LNCS, vol. 7636, pp. 703–717. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  6. 6.
    Smith, R.: Using games to improve productivity in software engineering. In: Proceedings of the ACM SIGKDD Workshop on Human Computation, HCOMP 2010 (2010)Google Scholar
  7. 7.
    Laredo, J., Vukovic, M., Rajagopal, S.: Service for Crowd-Driven Gathering of Non-Discoverable Knowledge. In: Pallis, G., et al. (eds.) ICSOC 2011 Workshops. LNCS, vol. 7221, pp. 283–294. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  8. 8.
    Dai, P., Mausam, Weld, D.S.: Decision-Theoretic Control of Crowd-Sourced Workflows. In: AAAI 2010 (2010)Google Scholar
  9. 9.
    Weld, D.S., Mausam, Dai, P.: Execution control for crowdsourcing. In: Proceedings of the 24th Annual ACM Symposium Adjunct on User Interface Software and Technology (UIST 2011 Adjunct), pp. 57–58. ACM, New York (2011)Google Scholar
  10. 10.
    Karger, D.R., Oh, S., Shah, D.: Iterative Learning for Reliable Crowdsourcing Systems. In: Neural Information Processing Systems, pp. 1953–1961 (2011)Google Scholar
  11. 11.
    Karger, D.R., Oh, S., Shah, D.: Budget-optimal Crowdsourcing Using Low-rank Matrix Approximations. In: 49th Annual Allerton Conference on Communication, Control, and Computing (Allerton), pp. 284–291 (2011)Google Scholar
  12. 12.
    Kittur, A., Smus, B., Khamkar, S., Kraut, R.E.: CrowdForge: crowdsourcing complex work. In: Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology (UIST 2011). ACM, New York (2011)Google Scholar
  13. 13.
    Mason, W., Watts, D.: Financial Incentives and the “Performance of Crowds”. In: Proceedings of ACK SIGKDD Workshop on Human Computation, HCOMP (2009)Google Scholar
  14. 14.
    Horton, J., Chilton, L.: The Labor Economics of Paid Crowdsourcing. In: Proceedings of the 11th ACM Conference on Electronic Commerce (2010)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Maja Vukovic
    • 1
  • Rajarshi Das
    • 1
  1. 1.IBM T.J. Watson Research CenterUSA

Personalised recommendations