Distributed Logistic Regression for Separated Massive Data

Abstract

In this paper, we study the distributed logistic regression to process the separated large scale data which is stored in different linked computers. Based on the Alternating Direction Method of Multipliers (ADMM) algorithm, we transform the solving of logistic problem into the multistep iteration process, and propose the distributed logistic algorithm which has controllable communication cost. Specifically, in each iteration of the distributed algorithm, each computer updates the local estimators and interacts the local estimators with the neighbors simultaneously. Then we prove the convergence of distributed logistic algorithm. Due to the decentralized property of computer network, the proposed distributed logistic algorithm is robust. The classification results of our distributed logistic method are same as the non-distributed approach. Numerical studies have shown that our approach are both effective and efficient which perform well in distributed massive data analysis.

Appendix

1.1 ADMM algorithm

Consider the following optimization problem

$$\begin{aligned} \underset{R, \theta }{\min }\ f(\theta )+h(R) \end{aligned}$$

(2.4)

$$s.t.\ MR+M^{'}\theta =0,$$

First, we form the quadratically augmented Lagrangian function

$$\begin{aligned} L(R,\theta ,V)=f(\theta )+h(R)+\langle V, MR+M^{'}\theta \rangle +\frac{c}{2}\Vert MR+M^{'}\theta \Vert _2^2, \end{aligned}$$

(2.5)

where $V:=\{V_{j}\}_{j\in \mathcal {C}}$ is Lagrange multipliers, and $c>0$ is a preselected penalty coefficient. Then we use ADMM algorithm to solve this problem.

ADMM algorithm entails three steps per iteration

Step 1: R updates,

$$\begin{aligned} R^{k+1}=\underset{R}{\arg \min }\ L(R,\theta ^{k},V^{k}). \end{aligned}$$

(A.1)

Step 2: $\theta$ updates,

$$\begin{aligned} \theta ^{k+1}=\underset{\theta }{\arg \min }\ L(R^{k+1},\theta ,V^{k}). \end{aligned}$$

(A.2)

Step 3: V updates,

$$\begin{aligned} V^{k+1}= V^{k}+c(MR^{k+1}+M^{'}\theta ^{k+1}). \end{aligned}$$

(A.3)

For details, in iteration $k+1$, we update $R^{k+1}$ via minimum $L(R,\theta ^{k},V^{k})$ with respect to R, update $\theta$ via minimum $L(R^{k+1},\theta ,V^{k})$ with respect to $\theta$, and update Lagrange multiplier via $V^{k+1}=V^{(k)}+c(MR^{(k+1)}+M^{'}\theta ^{(k+1)})$, until the algorithm converges.

1.2 Proof of Theorem 2

Proof

Theorem 2 can be considered as a special case of [20]. This paper analyzed the convergence of ADMM for minimizing possible nonconvex objective problem. It’s enough to show that our distributed optimization problem satisfies the convergence conditions A1-A5 in [20]. It’s obvious that A1 holds. Since both M and M$^{'}$ are full rank, then A2 and A5 are established. And $f(\theta )$ is Lipschitz differentiable with constant $L_f$, A4 holds. For any $j\in J$, and any $R_j$, $R_j^{'}$, (2.6) holds, thus A3 established. Then Theorem 2 has been proved.