Negotiation Strategy of Divisible Tasks for Large Dataset Processing

  • Quentin BaertEmail author
  • Anne-Cécile Caron
  • Maxime Morge
  • Jean-Christophe Routier
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10767)


MapReduce is a design pattern for processing large datasets on a cluster. Its performances depend on some data skews and on the runtime environment. In order to tackle these problems, we propose an adaptive multiagent system. The agents interact during the data processing and the dynamic task allocation is the outcome of negotiations. These negotiations aim at improving the workload partition among the nodes within a cluster and so decrease the runtime of the whole process. Moreover, since the negotiations are iterative the system is responsive in case of node performance variations. In this paper, we show how, when a task is divisible, an agent may split it in order to negotiate its subtasks.


Application of mas Automated negotiation Adaptation 


  1. 1.
    Dean, J., Ghemawat, S.: MapReduce: simplified data processing on large clusters. In: SOSDI, pp. 137–150 (2004)Google Scholar
  2. 2.
    Kwon, Y., Balazinska, M., Howe, B., Rolia, J.: Skewtune: mitigating skew in MapReduce applications. In: ACM SIGMOD ICMD, pp. 25–36 (2012)Google Scholar
  3. 3.
    Kwon, Y., Ren, K., Balazinska, M., Howe, B.: Managing skew in Hadoop. IEEE Data Eng. Bull. 36(1), 24–33 (2013)Google Scholar
  4. 4.
    Baert, Q., Caron, A.C., Morge, M., Routier, J.C.: Fair multi-agent task allocation for large datasets analysis. KAIS (2017). Scholar
  5. 5.
    Lama, P., Zhou, X.: Aroma: automated resource allocation and configuration of MapReduce environment in the cloud. In: ICAC, pp. 63–72 (2012)Google Scholar
  6. 6.
    Verma, A., Cherkasova, L., Campbell, R.: Aria: automatic resource inference and allocation for MapReduce environments. In: ICAC, pp. 235–244 (2011)Google Scholar
  7. 7.
    Chen, Q., Zhang, D., Guo, M., Deng, Q., Guo, S.: SAMR: a self-adaptive MapReduce scheduling algorithm in heterogeneous environment. In: ICCIT, pp. 2736–2743. IEEE (2010)Google Scholar
  8. 8.
    Liroz-Gistau, M., Akbarinia, R., Valduriez, P.: FP-Hadoop: efficient execution of parallel jobs over skewed data. VLDB Endow. 8(12), 1856–1859 (2015)CrossRefGoogle Scholar
  9. 9.
    Li, W.: Random texts exhibit Zipf’s-law-like word frequency distribution. IEEE Trans. Inf. Theory 38(6), 1842–1845 (1992)CrossRefGoogle Scholar
  10. 10.
    Lin, J.: The curse of Zipf and limits to parallelization: a look at the stragglers problem in MapReduce. In: Workshop on Large-Scale Distributed Systems for Information Retrieval (2009)Google Scholar
  11. 11.
    Gufler, B., Augsten, N., Reiser, A., Kemper, A.: Handling data skew in MapReduce. In: ICCCSS, pp. 574–583 (2011)Google Scholar
  12. 12.
    Vinyals, M., Macarthur, K.S., Farinelli, A., Ramchurn, S.D., Jennings, N.R.: A message-passing approach to decentralized parallel machine scheduling. Comput. J. 57(6), 856–874 (2014)CrossRefGoogle Scholar
  13. 13.
    Nongaillard, A., Mathieu, P.: Egalitarian negotiations in agent societies. AAI 25(9), 799–821 (2011)Google Scholar
  14. 14.
    Essa, Y.M., Attiya, G., El-Sayed, A.: Mobile agent based new framework for improving big data analysis. IJACSA 5(3), 25–32 (2014)Google Scholar
  15. 15.
    Gray, J., et al.: Data cube: a relational aggregation operator generalizing group-by, cross-tab, and sub-totals. Data Mining Knowl. Discov. 1(1), 29–53 (1997)CrossRefGoogle Scholar
  16. 16.
    Baert, Q., Caron, A.C., Morge, M., Routier, J.C.: Stratégie de découpe de tâche pour le traitement de données massives. In: Garbay, C., Bonnet, G., (eds.) Journées Francophones sur les Systèmes Multi-Agents. Cohésion: fondement ou propriété émergente, Caen, France, Cépaudès édition, pp. 65–75, July 2017Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Quentin Baert
    • 1
    Email author
  • Anne-Cécile Caron
    • 1
  • Maxime Morge
    • 1
  • Jean-Christophe Routier
    • 1
  1. 1.Univ. Lille, CNRS, Centrale Lille, UMR 9189 - CRIStAL - Centre de Recherche en Informatique Signal et Automatique de LilleLilleFrance

Personalised recommendations