LineChange: An Analytic Framework for Automated Moderation of Crowdsourcing Systems

  • Brent D. FegleyEmail author
  • Ryan Mullins
  • Ben Ford
  • Chad Weiss
Conference paper
Part of the Communications in Computer and Information Science book series (CCIS, volume 850)


If humans are more productive in collective problem-solving with a modicum of active help and guidance, then the potential of automated moderation of crowdsourcing systems has yet to be realized. Here, we present the conceptual design of an intelligent machine capable of (a) monitoring the temporal, structural, and emergent characteristics of participant behavior in a problem-solving process, and (b) modifying team structure and prompting participants for input at opportune or transitional moments in that process—by configuration, rule, or inference—to achieve collective goals and optimize output. The design is unique in treating teams as composable objects, in being scale-free, in relying on configuration and inference (not hard-coding), and in treating participant behaviors as sensory input.


Crowdsourcing Insourcing Network generation Automated facilitation Decision support 



This research was performed in connection with contract N68335-18-C-0040 with the U.S. Office of Naval Research. We would like to thank Dr. Yiling Chen and Dr. Predrag Neskovic for their contributions to this work as thought partners.


  1. 1.
    Anson, R., Bostrom, R., Wynne, B.: An experiment assessing group support system and facilitator effects on meeting outcomes. Manag. Sci. 41(2), 189–208 (1995). Scholar
  2. 2.
    Austen-Smith, D., Feddersen, T.: Deliberation and voting rules. In: Austen-Smith, D., Duggan, J. (eds.) Social Choice and Strategic Decisions: Essays in Honor of Jeffrey S. Banks. Studies in Choice and Welfare, pp. 269–316. Springer, Heidelberg (2005). Scholar
  3. 3.
    Bernstein, M.S., Little, G., Miller, R.C., Hartmann, B., Ackerman, M.S., Karger, D.R., Crowell, D., Panovich, K.: Soylent: a word processor with a crowd inside. Commun. ACM 58(8), 85–94 (2015). Scholar
  4. 4.
    Briggs, R.O., Reinig, B.A.: Bounded ideation theory. J. Manag. Inf. Syst. 27(1), 123–144 (2010). Scholar
  5. 5.
    Dennis, A.R., Williams, M.L.: Electronic brainstorming: theory, research, and future directions. In: Paulus, P.B., Nijstad, B.A. (eds.) Group Creativity: Innovation Through Collaboration, pp. 160–180. Oxford University Press, Oxford (2003). Scholar
  6. 6.
    Granovetter, M.: The strength of weak ties: a network theory revisited. Sociol. Theory 1, 201–233 (1983). Scholar
  7. 7.
    Guimer, R., Uzzi, B., Spiro, J., Amaral, L.A.N.: Team assembly mechanisms determine collaboration network structure and team performance. Science 308(5722), 697–702 (2005). Scholar
  8. 8.
    Judd, S., Kearns, M., Vorobeychik, Y.: Behavioral dynamics and influence in networked coloring and consensus. Proc. Nat. Acad. Sci. 107(34), 14978–14982 (2010). Scholar
  9. 9.
    Kim, T., Chang, A., Holland, L., Pentland, A.S.: Meeting mediator: enhancing group collaboration using sociometric feedback. In: Proceedings of the 2008 ACM Conference on Computer Supported Cooperative Work, CSCW 2008, pp. 457–466. ACM (2008).
  10. 10.
    Mao, A., Procaccia, A.D., Chen, Y.: Better human computation through principled voting. In: AAAI (2013)Google Scholar
  11. 11.
    Mellers, B., Stone, E., Murray, T., Minster, A., Rohrbaugh, N., Bishop, M., Chen, E., Baker, J., Hou, Y., Horowitz, M., Ungar, L., Tetlock, P.: Identifying and cultivating superforecasters as a method of improving probabilistic predictions. Perspect. Psychol. Sci. 10(3), 267–281 (2015). Scholar
  12. 12.
    Nemeth, C.J., Nemeth-Brown, B.: Better than individuals? The potential benefits of dissent and diversity for group creativity. In: Paulus, P.B., Nijstad, B.A. (eds.) Group Creativity: Innovation Through Collaboration, pp. 63–84. Oxford University Press, Oxford (2003). Scholar
  13. 13.
    Ottaviani, M., Srensen, P.: Information aggregation in debate: who should speak first? J. Public Econ. 81(3), 393–421 (2001). Scholar
  14. 14.
    Post, M., Callison-Burch, C., Osborne, M.: Constructing parallel corpora for six Indian languages via crowdsourcing. In: Proceedings of the Seventh Workshop on Statistical Machine Translation, pp. 401–409. Association for Computational Linguistics, Montréal, June 2012.
  15. 15.
    Spencer, D.: Card Sorting: Designing Usable Categories. Rosenfeld Media, New York (2009)Google Scholar
  16. 16.
    Wong, Z., Aiken, M.: Automated facilitation of electronic meetings. Inf. Manag. 41(2), 125–134 (2003). Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  • Brent D. Fegley
    • 1
    Email author
  • Ryan Mullins
    • 1
  • Ben Ford
    • 1
  • Chad Weiss
    • 1
  1. 1.Aptima, Inc.WoburnUSA

Personalised recommendations