Minimizing a sum of clipped convex functions

Abstract

We consider the problem of minimizing a sum of clipped convex functions. Applications of this problem include clipped empirical risk minimization and clipped control. While the problem of minimizing the sum of clipped convex functions is NP-hard, we present some heuristics for approximately solving instances of these problems. These heuristics can be used to find good, if not global, solutions, and appear to work well in practice. We also describe an alternative formulation, based on the perspective transformation, that makes the problem amenable to mixed-integer convex programming and yields computationally tractable lower bounds. We illustrate our heuristic methods by applying them to various examples and use the perspective transformation to certify that the solutions are relatively close to the global optimum. This paper is accompanied by an open-source implementation.

Appendices

Appendix A: Difference of convex formulation

In this section we make the observation that (1) can be expressed as a difference of convex (DC) programming problem.

Let $h_i(x) = \max (f_i(x) - \alpha _i, 0)$. This (convex) function measures how far $f_i(x)$ is above $\alpha _i$. We can express the ith term in the sum as

$$\begin{aligned} \min \{f_i(x), \alpha _i\} = f_i(x) - h_i(x), \end{aligned}$$

since when $f_i(x) \le \alpha _i$, we have $h_i(x)=0$, and when $f_i(x) > \alpha$, we have $h_i(x)=f_i(x)-\alpha _i$. Since $f_i$ and $h_i$ are convex, (1) can be expressed as the DC programming problem

$$\begin{aligned} \begin{array}{ll} \text{ minimize }&f_0(x) + \displaystyle \sum _{i=1}^m f_i(x) - \displaystyle \sum _{i=1}^m h_i(x), \end{array} \end{aligned}$$

(16)

with variable x. We can apply then well-known algorithms like the convex-concave procedure [24, 28] to (approximately) solve (16).

Appendix B: Minimal convex extension

If we replace each $f_i$ with any function ${\tilde{f}}_i$ such that ${\tilde{f}}_i(x) = f_i(x)$ when $f_i(x) \le \alpha _i$, we get an equivalent problem. One such ${\tilde{f}}_i$ is the minimal convex extension of $f_i$, which is given by

$$\begin{aligned} {\tilde{f}}_i(x) {:}{=}\sup \{f_i(z) + g^T (x-z) \mid g \in \partial f_i(z), f_i(z) \le \alpha _i, z\in {\text{ R }}^n\}. \end{aligned}$$

In general, the minimal convex extension of a function is often hard to compute, but it can be represented analytically in some (important) special cases. For example, if $f_i(x)=(a^T x - b)^2$, the minimal convex extension is the Huber penalty function, or

$$\begin{aligned} {\tilde{f}}_i(x) = {\left\{ \begin{array}{ll} (a^Tx - b)^2 &{} |a^Tx-b| \le \alpha _i, \\ \alpha _i (2|a^Tx-b| - \alpha _i) &{} \text {otherwise}. \end{array}\right. } \end{aligned}$$

Using the minimal convex extension leads to an equivalent problem, but, depending on the algorithm, replacing $f_i$ with ${\tilde{f}}_i$ can lead to better numerical performance.