Functional Federated Learning in Erlang (ffl-erl)

Abstract

The functional programming language Erlang is well-suited for concurrent and distributed applications, but numerical computing is not seen as one of its strengths. Yet, the recent introduction of Federated Learning, which leverages client devices for decentralized machine learning tasks, while a central server updates and distributes a global model, motivated us to explore how well Erlang is suited to that problem. We present the Federated Learning framework ffl-erl and evaluate it in two scenarios: one in which the entire system has been written in Erlang, and another in which Erlang is relegated to coordinating client processes that rely on performing numerical computations in the programming language C. There is a concurrent as well as a distributed implementation of each case. We show that Erlang incurs a performance penalty, but for certain use cases this may not be detrimental, considering the trade-off between speed of development (Erlang) versus performance (C). Thus, Erlang may be a viable alternative to C for some practical machine learning tasks.

A Mathematical Derivation of Federated Stochastic Gradient Descent

In Sect. 2.2 we briefly describe Federated Stochastic Gradient Descent. In the current section, we present the complete derivation. As a reminder, we stated that in Stochastic Gradient Descent, weights are updated this way:

$$\begin{aligned} w := w - \frac{\eta }{n} \displaystyle \sum _{i=1}^{n} \nabla F_i(w). \end{aligned}$$

(6)

Furthermore, we started with the following equation, which is the objective function we would like to minimize:

$$\begin{aligned} F(w) = \frac{1}{n} \displaystyle \sum _{j=1}^{k} |P_j| F^{j}(w). \end{aligned}$$

(7)

The gradient of $F^{j}$ is expressed in the following formula:

$$\begin{aligned} \nabla F^{j}(w) = \frac{1}{|P_j|} \displaystyle \sum _{i \in P_{j}}^{} \nabla F_i(w), j = 1,\cdots ,k. \end{aligned}$$

(8)

To continue from here, each client updates the weights of the machine learning model the following way:

$$\begin{aligned} w_j = w - \frac{\eta }{|P_j|} \displaystyle \sum _{i \in P_{j}}^{} \nabla F_i(w). \end{aligned}$$

(9)

On the server, the weights of the global model are updated. The original equation can be reformulated in a few steps:

$$\begin{aligned} w :=&\frac{1}{n}\left( \displaystyle \sum _{j=1}^{k} w_j |P_j|\right) \end{aligned}$$

(10)

$$\begin{aligned} =&\frac{1}{n}\displaystyle \sum _{j=1}^{k} \left( w - \frac{\eta }{|P_j|} \displaystyle \sum _{i \in P_{j}}^{} \nabla F_i(w)\right) |P_j| \end{aligned}$$

(11)

$$\begin{aligned} =&\frac{1}{n}\displaystyle \sum _{j=1}^{k} |P_j|w - \frac{1}{n}\eta \displaystyle \sum _{j=1}^{k} \displaystyle \sum _{i \in P_{j}}^{} \nabla F_i(w) \end{aligned}$$

(12)

$$\begin{aligned} =&w - \frac{\eta }{n} \displaystyle \sum _{i=1}^{n} \nabla F_i (w). \end{aligned}$$

(13)

The reformulation in the last line is equivalent to Eq. 6 above. In case the transformation between Eqs. 12 and 13 is unclear, consider that the summand simplifies to

$$\begin{aligned} \frac{1}{n}\displaystyle \sum _{j=1}^{k} |P_j|w = \frac{1}{n} n w = w. \end{aligned}$$

(14)

The second summand in Eq. 12 can be simplified as follows:

$$\begin{aligned} \frac{1}{n}\eta \displaystyle \sum _{j=1}^{k} \displaystyle \sum _{i \in P_{j}}^{} \nabla F_i(w) = \frac{1}{n}\eta \displaystyle \sum _{i = 1}^{n} \nabla F_i(w) = \frac{\eta }{n} \displaystyle \sum _{i = 1}^{n} \nabla F_i(w). \end{aligned}$$

(15)