Perturbed proximal primal–dual algorithm for nonconvex nonsmooth optimization

Abstract

In this paper, we propose a perturbed proximal primal–dual algorithm (PProx-PDA) for an important class of linearly constrained optimization problems, whose objective is the sum of smooth (possibly nonconvex) and convex (possibly nonsmooth) functions. This family of problems can be used to model many statistical and engineering applications, such as high-dimensional subspace estimation and the distributed machine learning. The proposed method is of the Uzawa type, in which a primal gradient descent step is performed followed by an (approximate) dual gradient ascent step. One distinctive feature of the proposed algorithm is that the primal and dual steps are both perturbed appropriately using past iterates so that a number of asymptotic convergence and rate of convergence results (to first-order stationary solutions) can be obtained. Finally, we conduct extensive numerical experiments to validate the effectiveness of the proposed algorithm.

Appendices

Appendix A

In this section, we justify Assumption [B4], which imposes the boundedness of the sequence of dual variables. Throughout this section we will assume that Assumptions A and [B1]–[B3] hold. First, we prove that when $\Vert \lambda ^{t+1}||\rightarrow \infty$, we have $\lim \inf _{r\rightarrow \infty } \frac{\beta ^{r+1}\Vert x^{r+1}-x^r\Vert }{\Vert \lambda ^{r+1}\Vert }= 0$. Using Assumption [B3] we have the following identity

$$\begin{aligned} \frac{\beta ^{r+1}\rho ^{r+1}}{2}\Vert x^{r+1}-x^r\Vert ^2 = \frac{(\beta ^{r+1})^2}{2 c_0}\Vert x^{r+1}-x^r\Vert ^2. \end{aligned}$$

(82)

Assume the contrary, that there exists $c_1>0$ such that

$$\begin{aligned} \lim _{r\rightarrow \infty }\,\, \beta ^{r+1}\Vert x^{r+1}-x^r\Vert ^2\ge \frac{c_1}{\beta ^{r+1}}\Vert \lambda ^{r+1}\Vert ^2. \end{aligned}$$

(83)

Then from (74), it is easy to show that when r is large enough, P is decreasing.

Similarly as in Lemma 3, it is relatively easy to show that the potential function is lower and upper bounded (the proof is included in Lemma 10–11 in the online version). The lower boundedness of the potential function and the fact that it is descending, implies that (75) holds true, which further implies that $\frac{1}{\beta ^{r+1}}\Vert \lambda ^{r+1}\Vert ^2\rightarrow 0$, according to (83). Examine the definition of the potential function in (73) and use the choice of c in (70) we conclude that $\frac{\beta ^{r+1}\rho ^{r+1}}{2}\Vert x^{r+1}-x^r\Vert ^2$ in the potential function is bounded. Therefore, there exists $D_1$ such that

$$\begin{aligned} \beta ^{r+1}\Vert x^{r+1}-x^r\Vert \le D_1. \end{aligned}$$

(84)

It follows that $c_1\Vert \lambda ^{r+1}\Vert ^2$ is also upper bounded. This contradicts our assumption that $\Vert \lambda ^{r}\Vert \rightarrow \infty$.

Next, we make use of some constraint qualification to argue the boundedness of the dual variables. The technique used in the proof is relatively standard, see recent works [21, 51]. Assume that the so-called Robinson’s condition is satisfied for problem (1) at ${\hat{x}}$ [62, Chap. 3]. This means $\{A d_x\mid d_x\in {\mathcal {T}}_{X}(\hat{x})\}={\mathbb {R}}^M,$ where $d_x$ is the tangent direction for convex set X, and ${\mathcal {T}}_{X}(\hat{x})$ is the tangent cone to the feasible set X at the point $\hat{x}$. Utilizing this assumption we will prove that the dual variable is bounded.

Lemma 6

Suppose the Robinson’s condition holds true for problem (1). Then the sequence of dual variable $\{\lambda ^r\}$ generated by (67b) is bounded.

Proof

We argue by contradiction. Suppose that the dual variable sequence is not bounded, i.e.,

$$\begin{aligned} \Vert \lambda ^r\Vert \rightarrow \infty . \end{aligned}$$

(85)

From the optimality condition of $x^{r+1}$ we have for all $x\in X$

$$\begin{aligned} \langle \nabla f(x^{r})+\xi ^{r+1}+A^T \lambda ^{r+1} + \beta ^{r+1}B^TB (x^{r+1}-x^r), x-x^{r+1}) \rangle \ge 0. \end{aligned}$$

Note that $\lim \inf _{r\rightarrow \infty } \frac{\beta ^{r+1}\Vert x^{r+1}-x^r\Vert }{\Vert \lambda ^{r+1}\Vert } =0$, so the following holds:

$$\begin{aligned} \lim \inf _{r\rightarrow \infty } \frac{\beta ^{r+1}\Vert B^T B(x^{r+1}-x^r)\Vert }{\Vert \lambda ^{r+1}\Vert } =0. \end{aligned}$$

Let us define a new bounded sequence as $\mu ^r = \lambda ^r/\Vert \lambda ^r\Vert , r=1,2, \ldots$. Let $(x^*, \mu ^*)$ be an accumulation point of $\{x^{r+1}, \mu ^{r+1}\}$. Assume that the Robinson’s condition holds at $x^*$. Dividing both sides of the above inequality by $\Vert \lambda ^{r+1}\Vert$ we obtain for all $x\in X$

$$\begin{aligned}&\langle \nabla f(x^{r})/\Vert \lambda ^{r+1}\Vert +\xi ^{r+1}/\Vert \lambda ^{r+1}\Vert +A^T \mu ^{r+1}\nonumber \\&\quad + \beta ^{r+1} B^T B (x^{r+1}-x^r)/\Vert \lambda ^{r+1}\Vert , x-x^{r+1} \rangle \ge 0. \end{aligned}$$

Taking the limit, passing a subsequence if necessary and utilizing the assumption that $\Vert \lambda ^{r+1}\Vert \rightarrow \infty$, and that X is a compact set, we obtain

$$\begin{aligned} \langle A^T \mu ^*, x-x^{*} \rangle \ge 0,\; \forall ~x\in X. \end{aligned}$$

Utilizing the Robinson’s condition, we know that there exists $x\in X$ and a scaling constant $c>0$ that such $cA(x-x^*) = - \mu ^*$, which combined with the above relation yields: $-c\Vert \mu ^*\Vert ^2\le 0$. Therefore we must have $\mu ^* = 0$. However, this contradicts to the fact that $\Vert \mu ^*\Vert =1$. Therefore, we conclude that $\{\lambda ^r\}$ is a bounded sequence.$\square$

Appendix B

We show how the sufficient conditions developed in “Appendix A” can be applied to the problems discussed in Sect. 1.2. We will focus on the partial consensus problem (10).

To proceed, we note that the Robinson’s condition reduces to the well-known Mangasarian–Fromovitz constraint qualification (MFCQ) if we set $X={\mathbb {R}}^{N}$, and write out explicitly the inequality constraints as $g(x)\le 0$ [62, Lemma 3.16]. To state the MFCQ, consider the following system

$$\begin{aligned} \quad&p_i(y)=0,\; i=1,\ldots , M\nonumber \\&g_j(y)\le 0,\; j=1,\ldots , P \end{aligned}$$

(86)

where $p_i:{\mathbb {R}}^N\rightarrow {\mathbb {R}}$ and $g_j:{\mathbb {R}}^N\rightarrow {\mathbb {R}}$ are all continuously differentiable functions. For a given feasible solution ${\hat{y}}$ let us use ${\mathcal {A}}({\hat{y}})$ to denote the indices for active inequality constraints, that is

$$\begin{aligned} {\mathcal {A}}({\hat{y}}):=\{1\le j\le P ~\mid ~ g_j({\hat{y}})=0\}. \end{aligned}$$

(87)

Let us define

$$\begin{aligned} p(y):=[p_1(y);p_2(y);\cdots ; p_M(y)],\quad g(y):=[g_1(y);g_2(y);\cdots ; g_P(y)]. \end{aligned}$$

Then the MFCQ holds for system (86) at point ${\hat{y}}$ if we have: 1) The rows of Jacobian matrix of p(y) denoted by $\nabla p({\hat{y}})$ are linearly independent. 2) There exists a vector $d_y\in {\mathbb {R}}^N$ such that

$$\begin{aligned} \nabla p({\hat{y}})d_y = 0, \quad \nabla g_j({\hat{y}})^Td_y<0, \; \forall ~ j\in {\mathcal {A}}({\hat{y}}). \end{aligned}$$

(88)

See [62, Lemma 3.17] for more details. In the following, we show that MFCQ holds true for problem (10) at any point (x, z) that satisfies $z\in Z$. Comparing the constraint set of this problem with system (86) we have the following specifications. The optimization variable $y=[x;z]$, where $x\in {\mathbb {R}}^N$ stacks all $x_i\in {\mathbb {R}}$ from N nodes (here we assume $x_i\in {\mathbb {R}}$ only for the ease of presentation). Also, $z\in {\mathbb {R}}^E$ stacks all $z_{e}\in {\mathbb {R}}$ for $e\in {\mathcal {E}}$. The equality constraint is written as $p(y)=[A,-I]y =0$, where $A\in {\mathbb {R}}^{E\times N}$ and I is an $E\times E$ identity matrix. Finally, for the inequality constraint we have $g_{e}(y)= |z_{e}| - \xi$, and the active set is given by ${\mathcal {A}}(y):={\mathcal {A}}^+(y)\cup {\mathcal {A}}^-(y)$, where

$$\begin{aligned} {\mathcal {A}}^+(y)=\{e\in {\mathcal {E}}\mid z_{e} = \xi \}, \quad {\mathcal {A}}^-(y)=\{e\in {\mathcal {E}}\mid z_{e} = -\xi \}. \end{aligned}$$

Without loss of generality we assume $\xi =1$. To show that MFCQ holds, consider a solution ${\hat{y}}:=({\hat{x}},{\hat{z}})$. First observe that the Jacobian of equality constraint is $\nabla p({\hat{y}})= [A,-I]$ which has full row rank. In order to verify the second condition we need to find a vector $d_y:=[d_x;d_z]\in {\mathbb {R}}^{N+E}$ such that

$$\begin{aligned}&Ad_x=d_z,\end{aligned}$$

(89a)

$$\begin{aligned}&[d_z]_{e}<0\quad \text {for} ~e\in {\mathcal {A}}^+({\hat{y}}) \end{aligned}$$

(89b)

$$\begin{aligned}&[d_z]_{e}>0\quad \text {for} ~e\in {\mathcal {A}}^-({\hat{y}}) \end{aligned}$$

(89c)

where $[d_z]_e$ denotes the eth component of vector $d_z$. Let us denote an all-one vector and all-zero vector by $\mathbf{1}$ and $\mathbf{0}$ respectively. To proceed, let us consider two different cases:

Case 1 For the vector ${\hat{z}}\in {\mathbb {R}}^E$ we have ${\hat{z}}\ne \mathbf{1}$ and ${\hat{z}}\ne -\mathbf{1}$. Let us take

$$\begin{aligned} d_z=\frac{1}{E}({\hat{z}}^T\mathbf{1})\mathbf{1} - {\hat{z}}. \end{aligned}$$

First we can show that $d_z\in \text {col}(A)$. Note that for our problem when the graph is connected, the only null space of A (which is the incidence of the graph) is spanned by the vector ${\mathbf {1}}$ [16]. Using this fact, we have $\mathbf{1}^Td_z = {\hat{z}}^T\mathbf{1} - \mathbf{1}^T{\hat{z}}=0$, therefore, $Ad_x=d_z$ holds true. Second, for $e\in {\mathcal {A}}^+({\hat{y}})$ we have that ${\hat{z}}_e=1$. Therefore, we can check that $[d_z]_e=\left[ \frac{1}{E}({\hat{z}}^T\mathbf{1})\mathbf{1} - {\hat{z}}\right] _e<0$, because $\frac{1}{E}({\hat{z}}^T\mathbf{1})\mathbf{1}<1$ from the fact that ${\hat{z}}\ne \mathbf{1}$. Condition (96b) is verified. Using similar argument we can verify condition (96c).

Case 2 Suppose we have ${\hat{z}}=\mathbf{1}$ (resp. ${\hat{z}}=-\mathbf{1}$). Since ${\hat{z}}\in \text {null}(A)$ let us set $d_x=\mathbf{0}$ and $d_z = -{\hat{z}}$ (resp. $d_z = {\hat{z}}$). First we have $Ad_x=d_z$. Second, for $e\in {\mathcal {A}}^+({\hat{y}})$ we have that $[d_z]_e<0$. Similarly, we have $[d_z]_e>0$ for $e\in {\mathcal {A}}^-({\hat{y}})$. All conditions (96a)–(96c) are verified. The above proof shows that MFCQ holds true for the sequence $\{(x^r,z^r)\}$ generated by the PProx-PDA algorithm, since in the algorithm it is always guaranteed that $z^r\in Z$.