Abstract
The communication between data-generating devices is partially responsible for a growing portion of the world’s power consumption. Thus reducing communication is vital, both, from an economical and an ecological perspective. For machine learning, on-device learning avoids sending raw data, which can reduce communication substantially. Furthermore, not centralizing the data protects privacy-sensitive data. However, most learning algorithms require hardware with high computation power and thus high energy consumption. In contrast, ultra-low-power processors, like FPGAs or micro-controllers, allow for energy-efficient learning of local models. Combined with communication-efficient distributed learning strategies, this reduces the overall energy consumption and enables applications that were yet impossible due to limited energy on local devices. The major challenge is then, that the low-power processors typically only have integer processing capabilities. This paper investigates an approach to communication-efficient on-device learning of integer exponential families that can be executed on low-power processors, is privacy-preserving, and effectively minimizes communication. The empirical evaluation shows that the approach can reach a model quality comparable to a centrally learned regular model with an order of magnitude less communication. Comparing the overall energy consumption, this reduces the required energy for solving the machine learning task by a significant amount.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Indeed, the average of two integers in binary representation can be computed using only the logical “and” and “or” \(+\) operations, as well as the bit-shift operator “\(>>\)” as \( \left\lfloor \frac{a+b}{2}\right\rfloor = (a \& b) + \left( (aXORb)>> 1\right) \).
- 2.
- 3.
- 4.
References
U.S. Energy Information Administration: How much electricity does a typical nuclear power plant generate? (2018). https://www.americangeosciences.org/critical-issues/faq/how-much-electricity-does-typical-nuclear-power-plant-generate. Accessed 02 Dec 2019
Asano, S., Maruyama, T., Yamaguchi, Y.: Performance comparison of FPGA, GPU and CPU in image processing. In: 2009 FPL, pp. 126–131. IEEE (2009)
Chow, C., Liu, C.: Approximating discrete probability distributions with dependence trees. IEEE Trans. Inf. Theor. 14(3), 462–467 (2006)
Dua, D., Graff, C.: UCI machine learning repository (2017). http://archive.ics.uci.edu/ml
Hammersley, J.M., Clifford, P.E.: Markov random fields on finite graphs and lattices. Unpublished manuscript (1971)
Kamp, M.: Black-box parallelization for machine learning. Ph.D. thesis, Rheinische Friedrich-Wilhelms-Universität Bonn (2019)
Kamp, M., et al.: Efficient decentralized deep learning by dynamic model averaging. In: Berlingerio, M., Bonchi, F., Gärtner, T., Hurley, N., Ifrim, G. (eds.) ECML PKDD 2018. LNCS (LNAI), vol. 11051, pp. 393–409. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-10925-7_24
Kamp, M., Boley, M., Keren, D., Schuster, A., Sharfman, I.: Communication-efficient distributed online prediction by dynamic model synchronization. In: Calders, T., Esposito, F., Hüllermeier, E., Meo, R. (eds.) ECML PKDD 2014. LNCS (LNAI), vol. 8724, pp. 623–639. Springer, Heidelberg (2014). https://doi.org/10.1007/978-3-662-44848-9_40
Kamp, M., Boley, M., Missura, O., Gärtner, T.: Effective parallelisation for machine learning. In: Advances in Neural Information Processing Systems, vol. 30, pp. 6477–6488. Curran Associates, Inc. (2017)
Kamp, M., Bothe, S., Boley, M., Mock, M.: Communication-efficient distributed online learning with kernels. In: Frasconi, P., Landwehr, N., Manco, G., Vreeken, J. (eds.) ECML PKDD 2016. LNCS (LNAI), vol. 9852, pp. 805–819. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46227-1_50
Kemp, S.: Digital 2019 global digital overview (2019). https://datareportal.com/reports/digital-2019-global-digital-overview. Accessed 02 Dec 2019
Lim, W.Y.B., et al.: Federated learning in mobile edge networks: a comprehensive survey. CoRR abs/1909.11875 (2019). http://arxiv.org/abs/1909.11875
McMahan, B., Moore, E., Ramage, D., Hampson, S., y Arcas, B.A.: Communication-efficient learning of deep networks from decentralized data. In: Artificial Intelligence and Statistics, pp. 1273–1282 (2017)
Mohan, N., Kangasharju, J.: Edge-fog cloud: a distributed cloud for internet of things computations. In: 2016 Cloudification of the Internet of Things (CIoT), pp. 1–6. IEEE (2016)
Piatkowski, N.: Distributed generative modelling with sub-linear communication overhead. In: Cellier, P., Driessens, K. (eds.) ECML PKDD 2019. CCIS, vol. 1167, pp. 281–292. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-43823-4_24
Piatkowski, N., Lee, S., Morik, K.: Integer undirected graphical models for resource-constrained systems. Neurocomputing 173, 9–23 (2016)
Piatkowski, N.P.: Exponential families on resource-constrained systems. Ph.D. thesis, TU Dortmund (2018)
Pihkola, H., Hongisto, M., Apilo, O., Lasanen, M.: Evaluating the energy consumption of mobile data transfer-from technology development to consumer behaviour and life cycle thinking. Sustainability 10(7), 2494 (2018)
Shi, W., Cao, J., Zhang, Q., Li, Y., Xu, L.: Edge computing: vision and challenges. IEEE Internet Things J. 3(5), 637–646 (2016)
Wainwright, M.J., Jordan, M.I.: Graphical models, exponential families, and variational inference. Found. Trends Mach. Learn. 1(1–2) (2008)
Acknowledgement
This research has been funded by the Federal Ministry of Education and Research of Germany as part of the competence center for machine learning ML2R (01\(\vert \)S18038A/B/C).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Heppe, L., Kamp, M., Adilova, L., Heinrich, D., Piatkowski, N., Morik, K. (2020). Resource-Constrained On-Device Learning by Dynamic Averaging. In: Koprinska, I., et al. ECML PKDD 2020 Workshops. ECML PKDD 2020. Communications in Computer and Information Science, vol 1323. Springer, Cham. https://doi.org/10.1007/978-3-030-65965-3_9
Download citation
DOI: https://doi.org/10.1007/978-3-030-65965-3_9
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-65964-6
Online ISBN: 978-3-030-65965-3
eBook Packages: Computer ScienceComputer Science (R0)