Smoothing gradient descent algorithm for the composite sparse optimization

1.   Introduction

Sparse optimization is a core problem of compressed sensing ^[1,2,3], signal and image processing ^[4,5,6,7], and high-dimensional statistical learning ^[4,8], etc. Sparsity is usually characterized by the $\ell_0$ norm, which is the cardinality of the support set of vector $\mathbf{x}\in\mathbb{R}^n$, denoted by $\|\mathbf{x}\|_0 = |supp(\mathbf{x})| = |\{i\in \{1, 2, \cdots, n\}:\mathbf{x}_i\neq 0 \}|$. The penalized formulation of sparse optimization can be expressed as the following cardinality regularized optimization:

where $f: \mathbb{R}^n\to \mathbb{R}$ is a loss function depending on the application, $\lambda > 0$ is the penalty parameter.

Compared with sparse optimization, composite sparse optimization problems enforce certain structural constraints instead of pure sparsity on the coefficients, which arise from many important applications in various fields, such as structural health monitoring ^[10], fault diagnosis ^[11], motion planning ^[12] and impact force identification^[13], etc. The most important method is to promote the sparsity of variables through linear transformation ^[14]. By imposing a regularization matrix $W = (W_1^{\top}, \ldots, W_p^{\top})^{\top}\in \mathbb{R}^{p\times n}$ on vector $\mathbf{x}$, composite sparse optimization is nicely encapsulated as the following optimization formulation:

A typical choice of function $f$ in problem (1.2) is the $\ell_2$ loss function given by $f(\mathbf{x}) = \frac{1}{2}\|A\mathbf{x}-b\|_2^2$, where $A = (A_1^{\top}, \ldots, A_m^{\top})^{\top}\in\mathbb{R}^{m\times n}$ and $b = (b_1, \ldots, b_m)^{\top}\in \mathbb{R}^m$, and the $\ell_1$ relaxation of the $\ell_0$ norm given by $\|W\mathbf{x}\|_1$, which was first defined and summarized as generalized LASSO with the general formulation of $W$ ^[15]. Unlike traditional LASSO, solving generalized LASSO efficiently on high- dimensional data is very challenging. A few attempts have been made to improve the efficiency of generalized LASSO, but this requires a specific form of the $W$ to work well ^[14,15,16], such as the fused LASSO problem ^[17], the TV regularizer ^[18] and trending filtering ^[19].

However, many loss functions of the composite sparse optimization problems cannot be expressed in the form of differentiable functions. The results in ^[20] showed that the least squares loss function can solve a class of linear regression problems but is not suitable for all types of data. We can choose the outlier that has a strong interference loss function, such as the ${\mathcal{\ell}_1}$ function, quantile regression function, or more general Huber class function. On the other hand, J. Fan et al. ^[20] pointed out that using the ${\mathcal{\ell}_1}$ relaxation often results in a biased estimator, various continuous nonconvex relaxation functions for ${\mathcal{\ell}_0}$ norm were proposed, such as the smoothly clipped absolute deviation (SCAD) function ^[20], the hard thresholding function ^[21], capped-${\mathcal{\ell}_1}$ function ^{[22,23,24,25,26]}, the transformed ${\mathcal{\ell}_1}$ function ^[27], etc. Here, we are interested in the capped-${\mathcal{\ell}_1}$ function as the relaxation function of the ${\mathcal{\ell}_0}$ norm, which is a simple relaxation function that satisfies specific properties. Z. Shen et al. ^[40] applied locally Lipschitz continuous scaled folded concave functions to the approximate ${\mathcal{\ell}_0}$ pseudo-norm. A generic nonsmooth but convex framework was established to gradually approximate the scaled folded concave functions. Numerical experimental results showed the proposed framework and algorithms admitted the exact sparsity-induced capability of the ${\mathcal{\ell}_0}$ pseudo-norm. Q. Chen et al.^[41] first explored using a class of locally Lipschitz scale folded concave functions to approach the ${\mathcal{\ell}_0}$. Then, a convex half-absolute method was proposed to precisely approximate these nonconvex nonsmooth functions. A double iterative algorithm was considered to solve the convex-relaxed composite optimization problems. Both ^[40] and ^[41] established a generic nonsmooth convex framework that gradually approximates these scale-folded concave functions based on the Legendre-Fenchel transformation to avoid directly solving a nonconvex optimization problem. However, we use a smoothing function for approximation to achieve the goal of solving the nonconvex optimization problem. The advantages of capped-${\mathcal{\ell}_1}$ function have been explored in various fields. For example, the authors in ^[28] put forward a capped ${\mathcal{\ell}_1}$-norm Sparse Representation method (CSR) for graph clustering. The proposed model learned the optimal graph for clustering by integrating sparse representation and capped ${\mathcal{\ell}_1}$-norm loss function. In order to utilize the advantages of the twin extreme learning machine and FDA, Z. Xue et al. ^[29] first put forward a novel classifier named Fisher-regularized twin extreme learning machine (FTELM). Also considering the instability of the ${\mathcal{\ell}_2}$-norm for the outliers, authors introduced the capped ${\mathcal{\ell}_1}$-norm into the FTELM model and proposed a more robust capped ${\mathcal{\ell}_1}$-norm FTELM (C${\mathcal{\ell}_1}$-FTELM) model. The capped ${\mathcal{\ell}_1}$ function was also discussed in the context of sparse group ${\mathcal{\ell}_0}$ regularized algorithms by ^[30]. It's worth noting that reference ^[9] gave an exact continuous relaxation problem with capped-${\mathcal{\ell}_1}$ penalty for nonsmooth convex loss function with cardinality penalty in the sense that both problems have the same optimal solution set. Moreover, a smoothing proximal gradient algorithm for finding a lifted stationary point of the continuous relaxation model was proposed. Regarding the solution of relaxation problems, T. Zhang ^[42] presented a multi-stage convex relaxation scheme for solving problems with non-convex objective functions. However, only parameter estimation performance was analyzed in ^[42]. Unfortunately, the result in ^[42] does not directly imply that multi-stage convex relaxation achieves unbiased recovery of the support set. H. Zhou et al. ^[43] proposed a new unified algorithm based on the local linear approximation (LLA) for maximizing the penalized likelihood for a broad class of concave penalty functions. It did not eliminate the bias issue. Here, we extend the results in ^[9] to composite sparse optimization and give a smoothing gradient descent algorithm for the continuous relaxation problem. The new algorithm exploits the piecewise linearity of the capped-${\mathcal{\ell}_1}$ penalty term in the relaxation problem. In view of the composite sparsity, if the subproblem in the algorithm does not have a closed solution, then our relaxation problem model analogizes the ${\mathcal{\ell}_1}$ penalty model for solving LASSO problems, using the smoothing gradient method to solve the subproblem. We prove that if the sequence generated by the algorithm has an accumulation point, then it is a lifted stationary point of relaxation problem.

In this paper, we consider the following composite sparse regression problem with cardinality penalty,

where $f: \mathbb{R}^n\to \mathbb{R}$ is a convex (not necessarily smooth) and bounded from below function, $\lambda$ is a positive parameter, and $\Omega = \{\mathbf{x}\in\mathbb{R}^n:l\leq W\mathbf{x}\leq u\}$. For example, the $\ell_1$ loss function given by

or the censored regression problem with

For a given parameter $\nu > 0$, the continuous relaxation of the $\ell_0$ penalty with the capped-$\ell_1$ function $\phi$ is given by

We consider the following continuous optimization problem to solve (1.3):

where $\Phi(W\mathbf{x}) = \sum_{i = 1}^p\phi(W_i\mathbf{x})$.

Composite sparse optimization has attracted much attention recently. In ^[15], a dual path algorithm was proposed for the generalized LASSO problem with any formulation of $W$. If the composite optimization problem is convex and the $W$ is the general linear map, one feasible approach is to apply the alternating direction method of multipliers (ADMM)^[31]. In ^[31], the author proposed a dual method for the variational problem in the form of $\inf\{f(Av)+g(v)\}$. Z. J. Bai ^[32] aimed to provide a coordinate gradient descent method with stepsize chosen by an Armijo-type rule to solve the problem $\min_{\mathbf{x}} f(\mathbf{x})+c\|L\mathbf{x}\|_1$ and $\min_{\mathbf{x}} \|A\mathbf{x}-b\|_2^2+c\|L\mathbf{x}\|_1$ efficiently, especially when the problems dimension is large.

In this paper, we use the exact continuous relaxation problem with capped-${\mathcal{\ell}_1}$ function to solve optimization problem (1.3) and present a smoothing gradient descent (SGD) algorithm. Since $W$ is a general linear mapping and $\Phi(W\mathbf{x})$ is an inseparable function, which makes the proximal gradient algorithm unable to be explicitly applied, we approximately solve the subproblem in the SGD algorithm. We prove that if there is an accumulation point, then the accumulation point is a lifted stationary point of (1.7).

Notation. We denote $\mathbb{N} = \{0, 1, \ldots\}$. For $\mathbf{x}\in \mathbb{R}^n$ and $\delta > 0$, let $\|\mathbf{x}\|: = \|\mathbf{x}\|_2$ and $\mathbb{B}_\delta(\mathbf{x})$ means the open ball centered at $\mathbf{x}$ with radius $\delta$. For a nonempty, closed, and convex set $\Omega\subseteq \mathbb{R}^n$, $N_\Omega(\mathbf{x})$ means the normal cone to $\Omega$ at $\mathbf{x}\in\Omega$. $\sigma_{\mathrm{min}}(W)$ is the minimum singular value of $W$. We denote $\|\mathbf{x}\|_\infty = \max\{|\mathbf{x}_1|, |\mathbf{x}_2|, \ldots, |\mathbf{x}_n|\}$.

2.   An exact continuous relaxation for problem (1.3)

Before starting this section, we first make the following two assumptions:

Assumption 2.1. Function $f$ is Lipschitz continuous on $\Omega$ with Lipschitz constant $L_f > 0$ and matrix $W$ has full column rank.

Assumption 2.2. Parameter $\nu$ in (1.6) satisfies $0 < \nu < \overline{\nu}: = \frac{\lambda\sigma_{\min}(W)}{L_f}$.

We suppose Assumptions 2.1 and 2.2 hold throughout the paper and assume that $L_f$ is large enough such that $L_f\geq\frac{\lambda\sigma_{\min}(W)}{\Gamma}$, where

When $f$ is defined by the ${\mathcal{\ell}_1}$ loss function (1.4) or the censored regression loss function (1.5), $L_f$ can be taken as $L_f = \|A\|_\infty$.

2.1.   Lifted stationary points of problem $(1.7)$

We first give the definition of lifted stationary points of (1.7) as that in ^[33]. Since function $\phi$ in (1.6) can be rephrased as

with $\theta_1(t) = 0$, $\theta_2(t) = \frac{t}{\nu}-1$ and $\theta_3(t) = -\frac{t}{\nu}-1$ for $t\in\mathbb{R}$, we denote

and

Definition 2.1. Point $\mathbf{x}\in\Omega$ is called a lifted stationary point of (1.7) if there exists $d = (d_1, \ldots, d_p)^{\top}\in\mathbb{D}^p(W\mathbf{x})$ such that

where

Under the definition of the range of $\nu$ in Assumption 2.2, we first prove that the element in $\mathbb{D}^p(W\mathbf{x})$ for a lifted stationary point satisfying (2.3) is unique.

Proposition 2.2. If $\bar{\mathbf{x}}$ is a lifted stationary point of (1.7), then the vector $d^{W\bar{\mathbf{x}}} = (d_1^{W\bar{\mathbf{x}}}, \ldots, d_p^{W\bar{\mathbf{x}}})^{\top}\in \mathbb{D}^p(W\bar{\mathbf{x}})$ satisfying (2.3) is unique. In particular, for $i = 1, \ldots, p,$

Proof. If $|W_i\bar{\mathbf{x}}|\neq\nu$, the statement is clearly valid. Hence, we only need to consider the case $|W_i\bar{\mathbf{x}}| = \nu$. When $W_i\bar{\mathbf{x}} = \nu$, since $\mathbb{D}(W_i\bar{\mathbf{x}}) = \{1, 2\},$ arguing by contradiction, we assume (2.3) holds with $d_i^{W{\bar{\mathbf{x}}}} = 1$, so $\vartheta^i(\bar{\mathbf{x}}) = 1$. By $\nu < \bar{\nu},$ we have $W_i\bar{\mathbf{x}}\in(l_i, u_i)$, and by (2.3), there exists $\xi(\bar{\mathbf{x}})\in\partial f(\bar{\mathbf{x}})$ such that

where $\vartheta^i(\bar{\mathbf{x}})$ is defined as (2.4).

It is easy to observe that the following relation holds

In fact, from the definition of the minimum singular value of $W$,

we have

Then, we see that

From the definition of $\vartheta^i(\bar{\mathbf{x}})$ (2.4), this yields that

where $W_I$ is the submatrix consisting of the rows in $W$ indexed by $I: = \{i:|W_i\bar{\mathbf{x}}|\leq\nu\}$ ^[34].

Combining (2.6) and (2.7), we have

This leads to a contradiction to $\nu < \frac{\lambda\sigma_{\min}(W)}{L_f}$. Then, (2.5) holds for $W_i\bar{\mathbf{x}} = \nu$. Similar analysis can be given for the case that $W_i\bar{\mathbf{x}} = -\nu$, which completes the proof. □

For a given $d = (d_1, \ldots, d_p)^{\top}\in\mathbb{D}^p: = \{d\in\mathbb{R}^p:d_i\in\{1, 2, 3\}, i = 1, \cdots, p\}$, we define

which is convex with respect to $\mathbf{x}$. It can be verified that

In particular, for a fixed $\bar{\mathbf{x}}\in\Omega$, $\Phi(W\bar{\mathbf{x}}) = \Phi^{d^{W\bar{\mathbf{x}}}}(W\bar{\mathbf{x}})$ and the following relation holds

Next lemma describes a lower bound property.

Lemma 2.3. If $\bar{\mathbf{x}}\in\Omega$ is a lifted stationary point of (1.7), then it holds that

Proof. Suppose that $\bar{\mathbf{x}}$ is a lifted stationary point of (1.7). Now we assume that $W_i\bar{\mathbf{x}}\in(-\nu, \nu)\setminus\{0\}$ for some $i\in{1, \ldots, p}$. So from (2.5) and Assumption 2.1, we have $d_i^{W\bar{\mathbf{x}}} = 1$ and $W_i\bar{\mathbf{x}}\in(l_i, u_i)$. By (2.3), there exists $\xi(\bar{\mathbf{x}})\in\partial f(\bar{\mathbf{x}})$. We have

Through the analysis in the proof of Proposition 2.2, combining (2.6) and (2.7), we have

which leads to a contradiction to $\nu < \frac{\lambda\sigma_{\min}(W)}{L_f}$. Consequently, $W_i\bar{\mathbf{x}}\in(-\nu, \nu)$ implies $W_i\bar{\mathbf{x}} = 0$ for $i\in{1, \ldots, p}$ {and} the proof is completed. □

2.2.   The relationship between problems $(1.3)$ and $(1.7)$

This subsection discusses the relationship between the global minimizers and local minimizers of (1.3) and (1.7). First, Theorem 2.4 discusses the relationship between the local minimizers of (1.3) and (1.7). Second, Theorem 2.5 states that (1.3) and (1.7) have the same global minimizers. We use the lower bound property mentioned in Lemma 2.3 to prove Theorems 2.4 and 2.5.

Theorem 2.4. If $\bar{\mathbf{x}}$ is a lifted stationary point of (1.7), then it is a local minimizer of (1.3) and the objective functions have the same value at $\bar{\mathbf{x}}$, i.e., $f(\bar{\mathbf{x}})+\lambda\Phi(W\bar{\mathbf{x}}) = f(\bar{\mathbf{x}})+\lambda\|W\bar{\mathbf{x}}\|_0$.

Proof. Combining the lower bound property of $W\bar{\mathbf{x}}$ in (2.10) and the definition of $\Phi^{d^{W\bar{\mathbf{x}}}}$ defined in (2.8), for any $\mathbf{x}\in\mathbb{R}^n$, we have

Then

Combining this with $\Phi(W\bar{\mathbf{x}}) = \|W\bar{\mathbf{x}}\|_0$ and (2.9), we have

Thus, $\bar{\mathbf{x}}$ is a local minimizer of (1.7). □

Theorem 2.4 indicates that any lifted stationary point of (1.7) is a local minimizer of (1.3), which means that any local minimizer of (1.7) is also certainly a local minimizer of (1.3).

Theorem 2.5. If $\bar{\mathbf{x}}\in\Omega$ is a global minimizer of (1.3) if and only if it is a global minimizer of (1.7). Moreover, problems (1.3) and (1.7) have the same optimal value.

Proof. On the one hand, let $\bar{\mathbf{x}}\in\Omega$ be a global minimizer of (1.7), and according to Definition 2.1, then we can obtain that $\bar{\mathbf{x}}$ is a lifted stationary point of (1.7). By (2.10), from $W_i\bar{\mathbf{x}}\in(-\nu, \nu)$, then $W_i\bar{\mathbf{x}} = 0$, so it gives $\Phi(W\bar{\mathbf{x}}) = \|W\bar{\mathbf{x}}\|_0$. We have

where the last inequality uses $\Phi(W\mathbf{x})\leq\|W\mathbf{x}\|_0$. Therefore, $\bar{\mathbf{x}}$ is a global minimizer of (1.3).

On the other hand, let $\bar{\mathbf{x}}\in\Omega$ be a global minimizer of (1.3). Assume on the contrary $\bar{\mathbf{x}}$ is not a solution of (1.7). Let $\hat{\mathbf{x}}$ be a global minimizer of (1.7), we obtain

From

we have

This contradicts the global optimality of $\bar{\mathbf{x}}$ for (1.3). Hence $\bar{\mathbf{x}}$ is a global minimizer of (1.7). Therefore, (1.3) and (1.7) have the same global minimizers and optimal values. □

When $f$ is convex, $\bar{\mathbf{x}}$ is a local minimizer of (1.3) if and only if $\bar{\mathbf{x}}\in\Omega$ satisfies

which is often used as a criterion for the local minimizers of problem (1.3).

Definition 2.6. We call $\bar{\mathbf{x}}\in\Omega$ a $\nu$-strong local minimizer of (1.3), if there exists $\overline{\xi}\in\partial f(\bar{\mathbf{x}})$ and $\overline{\eta}\in N_\Omega(\bar{\mathbf{x}})$ such that for any $i\in supp(W\bar{\mathbf{x}})$, it holds

By (2.12), any $\nu$-strong local minimizer of (1.3) is a local minimizer of it. Below we provide a result on the relationship between the $\nu$-strong local minimizers of (1.3) and the lifted stationary points of (1.7).

Proposition 2.7. $\bar{\mathbf{x}}\in\Omega$ is a $\nu$-strong local minimizer of (1.3) if and only if it is a lifted stationary point of (1.7).

Proof. First, by (2.9), we see that if $\bar{\mathbf{x}}$ is a lifted stationary point of (1.7), then

Combining the Lemma 2.3 and $\Phi^{d^{W\bar{\mathbf{x}}}}(W\mathbf{x})\leq\|W\mathbf{x}\|_0, \; \forall \mathbf{x}\in\mathbb{B}_\varrho(\bar{\mathbf{x}}), \; \varrho > 0$ in (2.11), then we have

so $\bar{\mathbf{x}}$ is a $\nu$-strong local minimizer of (1.3).

Next, because $\bar{\mathbf{x}}$ is a $\nu$-strong local minimizer of (1.3), it is also a local minimizer of (1.3), suppose $\bar{\mathbf{x}}$ is a local minimizer of (1.3) but not a local minimizer of (1.7). Then there exists a local minimizer of (1.7) denoted by $\hat{\mathbf{x}}$, combining (2.9), $\Phi^{d^{W\hat{\mathbf{x}}}}(W\bar{\mathbf{x}})\leq\|W\bar{\mathbf{x}}\|_0, \; \forall \bar{\mathbf{x}}\in\mathbb{B}_\varrho(\hat{\mathbf{x}})$ in (2.11) and $\Phi(W\hat{\mathbf{x}}) = \|W\hat{\mathbf{x}}\|_0$, we have

which leads to a contradiction. Thus, the local minimizer of (1.3) is the local minimizer of (1.7), that is to say, the $\nu$-strong local minimizer of (1.3) is the lifted stationary point of (1.7). □

We use Table 1 to clearly demonstrate the relationship between (1.3) and (1.7).

3.   Smoothing gradient descent algorithm (SGD)

The main content of this section is to find the lifted stationary point of (1.7). Due to the existence of matrix $W$, we cannot express the explicit solution using the proximal gradient method. We first approximate $f$ by a smoothing function and propose some preliminary theories on the smoothing methods; the second section proposes our algorithm; and the third section conducts a convergence analysis on the proposed algorithm for solving (1.7).

3.1.   Smoothing approximation method

Throughout this paper, we approximate the loss function $f$ by a smoothing function $\tilde{f}$ in (1.7). When it is clear from the context, the derivative of $\tilde{f}(\mathbf{x}, \mu)$ with respect to $\mathbf{x}$ is simply denoted as $\nabla\tilde{f}(\mathbf{x}, \mu)$. We denote

where smoothing parameter $\mu > 0$ and $d\in\mathbb{D}^p$. For any fixed $\mu > 0$ and $d\in\mathbb{D}^p$, $\widetilde{\mathcal{W}}^d(\mathbf{x}, \mu)$ is nonsmooth and convex, and $\widetilde{\mathcal{W}}(\mathbf{x}, \mu)$ is nonsmooth and nonconvex. Due to

we obtain

The following definition describes some theories about the smoothing function $\tilde{f}$, which is frequently used in the proof of convergence analysis.

Definition 3.1 We call $\tilde{f}:\mathbb{R}^n\times[0, \bar{\mu}]\rightarrow \mathbb{R}$ with $\bar{\mu} > 0$ a smoothing function of the convex function $f$ in (1.7), if $\tilde{f}(\cdot, \mu)$ is continuously differentiable in $\mathbb{R}^n$ for any fixed $\mu > 0$ and satisfies the following conditions:

(ⅰ) $\lim_{\mathbf{z}\rightarrow \mathbf{x}, \mu\downarrow0}\tilde{f}(\mathbf{z}, \mu) = f(\mathbf{x}), \forall \mathbf{x}\in\Omega;$

(ⅱ) (convexity) $\tilde{f}(\mathbf{x}, \mu)$ is convex with respect to $\mathbf{x}$ in $\Omega$ for any fixed $\mu > 0$;

(ⅲ) (gradient consistency) $\{\lim_{\mathbf{z}\rightarrow \mathbf{x}, \mu\downarrow0}\bigtriangledown_\mathbf{z}\tilde{f}(\mathbf{z}, \mu)\}\subseteq\partial f(\mathbf{x}), \forall \mathbf{x}\in\Omega;$

(ⅳ) (Lipschitz continuity with respect to $\mu$) there exists a positive constant $\kappa$ such that

(ⅴ) (Lipschitz continuity with respect to $\mathbf{x}$) there exists a constant $L > 0$ such that for any $\mu\in(0, \bar{\mu}], \bigtriangledown_\mathbf{x}\tilde{f}(\cdot, \mu)$ is Lipschitz continuous on $\Omega$ with Lipschitz constant $L\mu^{-1}.$

By virtue of Definition 3.1-(ⅳ), we obtain that

Next, we aim to solve the following problem with $\mu > 0$ and vector $d\in\mathbb{D}^p$

by introducing an approximation of $\widetilde{\mathcal{W}}^d(\cdot, \mu)$ around a given point $\mathbf{z}$ as follows:

with a constant $\gamma > 0$. $\Phi^d(W\mathbf{x})$ is convex with respect to $\mathbf{x}$ for any fixed $d\in\mathbb{D}^p$, function $G_{d, \gamma}(\mathbf{x}, \mathbf{z}, \mu)$ is a strongly convex function with respect to $\mathbf{x}$ for any fixed $d, \gamma, \mathbf{z}$ and $\mu$. Then, we solve the following problem

to find the approximate solution of (3.3).

3.2.   Smoothing gradient descent algorithm

In this subsection, we propose a new algorithm (see Algorithm 1) for finding a lifted stationary point of (1.7). Specially, since $W$ is a general linear mapping and $\Phi(W\mathbf{x})$ is an inseparable function, which makes the smoothing proximal gradient algorithm ^[9] cannot explicitly solve a subproblem. The proposed algorithm combines the smoothing method and the smoothing gradient descent algorithm, so we call it the smoothing gradient descent (SGD) algorithm. We use the SGD algorithm to obtain approximate solutions of the subproblem. Let

and denote $p_r^s$ the $r$th smallest number in $\mathcal{P}^s$. Then, we can obtain the following updating method of $\{\mu_k\}$

which will be used in the proof of Lemmas 3.2 and 3.4.

3.3.   Convergence analysis

Lemma 3.2. The proposed SGD algorithm is well-defined, and the sequences $\{\mathbf{x}^k\}$, $\{\gamma^k\}$ and $\{\mu_k\}$ generated by it own the following properties:

(ⅰ) $\{\mathbf{x}^k\}\subseteq\Omega$ and $\{\gamma_k\}\subseteq[\underline{\gamma}, \max\{\overline{\gamma}, \rho L\}];$

(ⅱ) there are infinite elements in $\mathcal{P}^s$ and $\lim_{k\rightarrow \infty}\mu_k = 0.$

Proof. (ⅰ) By organizing (3.7), we can obtain

According to Definition 3.1-(ⅴ), (3.7) holds when $\gamma_k\geq L$. Thus the updating of $\gamma_k$ in Step 2 is at most $\log_\rho(\frac{L}{\underline{\gamma}})+1$ times at each iteration. Hence, the SGD algorithm is well-defined, and we have that $\gamma_k\leq\max\{\overline{\gamma}, \rho L\}, \forall k\in\mathbb{N}.$ From (3.8), it is easy to verify that $\mathbf{x}^{k+1}\in\Omega$ by $\mathbf{x}^{k}\in\Omega$ and $\hat{\mathbf{x}}^{k+1}\in\Omega$.

(ⅱ) Since $\{\mu_k\}$ is non-increasing, to prove (ⅱ), we assume that $\lim_{k\rightarrow \infty}\mu_k = \hat{\mu} > 0$ by contradiction. If $\{\mu_k\}$ converges to a non-zero value, then the iteration of (3.10) is finite, which means that there exists $\mathrm{K}\in\mathbb{N}$ such that $\mu_k = \hat{\mu}, \forall k\geq\mathrm{K}$. Substituting $\hat{\mu}$ into (3.9), we obtain

By the above inequality, we have

However, by $\{\mathbf{x}^k\}\subseteq\Omega$, (3.2) and Theorem 2.5, then

(3.11) and (3.12) are contradictory; (ⅱ) holds. □

Lemma 3.3. For any $k\in\mathbb{N}$, we have

which implies $\{\widetilde{\mathcal{W}}(\mathbf{x}^{k+1}, \mu_k)+\kappa\mu_k\}$ is non-increasing and $\lim\limits_{k\rightarrow \infty}\widetilde{\mathcal{W}}(\mathbf{x}^{k+1}, \mu_k) = \lim\limits_{k\rightarrow \infty}\mathcal{W}(\mathbf{x}^{k}).$

Proof. Since $G_{d^k, \gamma_k}(\mathbf{x}, \mathbf{x}^k, \mu_k)$ is strongly convex with modulus $\gamma_k\mu_k^{-1},$ we have

using the definition of $\hat{\mathbf{x}}^{k+1}$ in (3.6) and $\mathbf{x}^{k+1} = \hat{\mathbf{x}}^{k+1}$ when (3.7) holds, we obtain

By the definition of function $G_{d^k, \gamma_k}$ given in (3.4), organizing the inequalities above, we have

Moreover, (3.7) can be written as

Summing up (3.15) and (3.16), we obtain that

For a fixed $\mu > 0$, the convexity of $\tilde{f}(\mathbf{x}, \mu)$ with respect to $\mathbf{x}$ indicates

Combining (3.17) and (3.18) and utilizing the definition of $\widetilde{\mathcal{W}}^{d^k}$, one has

Letting $\mathbf{x} = \mathbf{x}^k$ in (3.19) and by $d^k = d^{{W\mathbf{x}}^k}$, we obtain

Because $\widetilde{\mathcal{W}}^{d^k}(\mathbf{x}^{k+1}, \mu_k)\geq\widetilde{\mathcal{W}}(\mathbf{x}^{k+1}, \mu_k),$ (3.20) leads to (3.13). Due to Definition 3.1-(iv), we have

then, it follows that

by (3.13), we obtain

(3.21) implies the non-increasing property of $\{\widetilde{\mathcal{W}}(\mathbf{x}^{k+1}, \mu_k)+\kappa\mu_k\}.$ This result and (3.12) ensure the existence of $\lim_{k\rightarrow \infty}\widetilde{\mathcal{W}}(\mathbf{x}^{k+1}, \mu_k)+\kappa\mu_k$. By virtue of $\lim_{k\rightarrow \infty}\mu_k = 0$ and Definition 3.1-(i), we obtain

The proof is completed. □

Lemma 3.4. The following statements hold:

(ⅰ) $\sum_{k = 0}^\infty\gamma_k\mu_k^{-1}\|\mathbf{x}^{k+1}-\mathbf{x}^k\|^2\leq2(\mathcal{W}(\mathbf{x}^0, \mu_{-1})+\kappa\mu_{-1}-\min_\Omega\mathcal{W});$

(ⅱ) $\sum_{k = 0}^\infty\mu_k^2\leq\Lambda$ with $\Lambda = \frac{1}{\alpha}(\widetilde{\mathcal{W}}(\mathbf{x}^0, \mu_{-1})+\kappa\mu_{-1}-\min_{\mathbf{x}\in\Omega}\mathcal{W}(\mathbf{x}))+\frac{2\mu_0^2\sigma}{2\sigma-1} < \infty$;

Proof. (ⅰ) Recalling (3.21), for all $k\in\mathbb{N}$, we obtain

Now adding up the above inequality over $k = 0, \ldots, \mathrm{K}$, it gives

By letting $\mathrm{K}$ in (3.22) tend to infinity and along with (3.12), we obtain (ⅰ).

(ⅱ) From (3.5), we have

where $p_r^s$ is the $r$th smallest element in $\mathcal{P}^s$. When $k\notin\mathcal{P}^s$, (3.9) gives

which together with the non-increasing property of $\{\widetilde{\mathcal{W}}(\mathbf{x}^{k+1}, \mu_k)+\kappa\mu_k\}$ and (3.12) implies

Combining (3.24) and (3.25), the proof of (ii) is completed. □

Theorem 3.5. If there is an accumulation point in $\{\mathbf{x}^k:k\in\mathcal{P}^s\}$, then the accumulation point is a lifted stationary point of (1.7).

Proof. Since (3.9) fails for $k\in\mathcal{P}^s$, by rearranging (3.21), we obtain that

which gives

Thus,

which together with $\lim_{k\rightarrow \infty}\mu_{k} = 0$ and $\{\gamma_{k}\}\subseteq[\underline{\gamma}, \max\{\bar{\gamma}, \rho L\}]$ implies

Let $\bar{\mathbf{x}}$ be an accumulation point of $\{\mathbf{x}^k\}_{k\in\mathcal{P}^s}$, (3.26) indicates that $\{\mathbf{x}^k\}$ exists a subsequence $\{\mathbf{x}^{k_t}\}_{k_t\in\mathcal{P}^s}$ converges to $\bar{\mathbf{x}}$. Similar analysis can be given for the case that $k_t\in\mathcal{P}^s$ implies

Recalling $\mathbf{x}^{k_t+1} = \hat{\mathbf{x}}^{k_t+1}$ defined in (3.6) and by its first-order necessary optimality condition, we have

Since the elements in $\{d^{k_t}:t\in\mathbb{N}\}$ are finite and $\lim_{t\rightarrow \infty}\mathbf{x}^{k_t+1} = \bar{\mathbf{x}}$, there exists a subsequence of $\{k_t\}$, denoted as $\{k_{t_j}\}$, and $\bar{d}\in\mathbb{D}^p(W\bar{\mathbf{x}})$ such that $d^{k_{t_j}} = \bar{d}, \forall j\in\mathbb{N}$. By the upper semicontinuity of $\partial\Phi^{\bar{d}}$ and $\lim_{j\rightarrow \infty}\mathbf{x}^{k_{t_j}+1} = \bar{\mathbf{x}}$, it gives

Along with the subsequence $\{k_{t_j}\}$ and letting $j\rightarrow \infty$ in (3.28), from Definition 3.1-(ⅲ), (3.27) and (3.29), we obtain that there exist $\bar{\xi}\in\partial f(\bar{\mathbf{x}})$ and $\bar{\zeta}^{\bar{d}}\in\partial\Phi^{\bar{d}}(W\bar{\mathbf{x}})$ such that

By $\bar{d}\in\mathbb{D}^p(W\bar{\mathbf{x}})$, thanks to the convexity of $f+\lambda\Phi^{\bar{d}}$, (3.30) implies

which implies that $\bar{\mathbf{x}}$ is a lifted stationary point of (1.7). □

4.   Numerical experiments

The purpose of this part is to test and verify the theoretical results and the properties of the SGD algorithm by the numerical experiments. We present Examples 1 and 2, which are respectively an under-determined linear regression problem and an over-determined censored regression problem. Especially, the process of solving subproblem (3.6) is very similar to the algorithm process of solving the LASSO problem.

All experiments are performed in MATLAB 2016a on a Lenovo PC with an Intel(R) Core(TM) i5-8250U CPU @1.60GHz 1801 Mhz and 8GB RAM. In the following examples, stopping criterion is set as

We stop the proposed algorithm if the number of iterations exceeds $\mathbf{\mathtt{Maxiter}}$ or the smoothing parameter is less than $\varepsilon$. Denote $\bar{\mathbf{x}}$ the output of iterate $\mathbf{x}^k$. Set the fixed parameter $\alpha = 1$ throughout the numerical experiments.

Example 4.1. (Linear regression problem) Linear regression problems have been widely used in information theory^[1], signal processing ^[35,36] and image restoration^[6,36]. As pointed out in ^[20], $\mathcal{\ell}_1$ loss function is nonsmooth, but more robust and has stronger capability of outlier-resistance than the least squares loss function in the linear regression problems. Then we consider the following $\mathcal{\ell}_0$ regularized linear regression problem with $\mathcal{\ell}_1$ loss function:

where $A\in \mathbb{R}^{m\times n}$ with $m = n$, $b\in\mathbb{R}^m$. A smoothing function of the $\mathcal{\ell}_1$ loss function can be defined by

Denote $s$ the $\mathcal{\ell}_0$ norm of true solution $\mathbf{x}^*$, i.e., $\|W\mathbf{x}^*\|_0 = s$. For the given positive integers $m, n$ and $s$, the data are generated by

In the algorithm, we set the parameters as below: $\underline{\gamma} = \overline{\gamma} = 1, \; \mu_0 = 3.533, \; \mathbf{\mathtt{Maxiter}} = 10^3, \; \nu = 35.6014, \; \sigma = 3.0003, \; \rho = 1.0001, \; \kappa = \frac{1}{2}.$ Generate $A, b$ and $\mathbf{x}^*$ with $m = n = 45$, $p = 45$ and $s = 2$, set $\lambda = 10^{-3}$ in (4.2) and $\varepsilon = 10^{-3}$ in the stopping criterion (4.1). We set $\mathbf{x}_0$ = $ones(n, 1)$. Figure 1 shows the numerical results. Figure 1 plots $\mathbf{x}^*$ and $\overline{\mathbf{x}}$, where $\mathbf{x}^*$ and $\overline{\mathbf{x}}$ denote the original signal (which can also be expressed as true solution) and the output of iterate $\mathbf{x}^k$ from the SGD algorithm. From Figure 1, we can see that the output of $\mathbf{x}^k$ is very close to the original generated signal.

Now we use another form of matrix $W$ to solve Example 4.1:

Set $\underline{\gamma} = \overline{\gamma} = 1, \; \mu_0 = 3.533, \; \mathbf{\mathtt{Maxiter}} = 10^3, \; \nu = 36, \; \sigma = 7, \; \rho = 1.0001 \; \mathrm{and}\; \kappa = \frac{1}{2}.$ We randomly generate the data as follows:

We run numerical experiments with $(m, n, p, s)$ = (45, 45, 45, 2). Set $\lambda = 10^{-3}$ in (4.2) and $\varepsilon = 10^{-3}$ in the stopping criterion (4.1). We define $\mathbf{x}_0$ = $randn(n, 1)$. From Figure 2, we can see that the output of $\mathbf{x}^k$ obtained by the SGD algorithm is also close to the true solution $\mathbf{x}^*$.

The last special case of W is the penalty matrix in 1-dimensional Fused LASSO ^[39]:

Set $\nu = 40, \; p = 45, \; n = 46$ and $\mathbf{x}_0$ = $ones(n, 1)$. The remaining parameter values are the same as the previous situation (see Figure 3). From Figures 1–3, we can see that the output of $\mathbf{x}^k$ obtained by the SGD algorithm is close to the true solution $\mathbf{x}^*$.

Example 4.2 (Censored regression problem) The application of censored regression problems has been studied in machine learning^[37], economics^[38], biomedical, and technical systems. The following censored regression problem is also a typical class of composite sparse optimization problems with nonsmooth convex loss functions. Now we consider the following ${\mathcal{\ell}_0}$ regularized censored regression problem:

where $A\in\mathbb{R}^{m\times n}$ and $b\in\mathbb{R}^m$. For the loss function in (1.5), a smoothing function of it can be defined by

Set $\varepsilon = 10^{-2}$, $\nu = 16.0009$, $\lambda = 10^{-3}$, $\mu_0 = 10.8999$, $\sigma = 4.0003$, $\kappa = \frac{1}{2}$, $\rho = 1.2006$ and $\mathbf{x}_0$ = $randn(n, 1)$. In this example, we run numerical experiments with $(m, n, p, s)$ = (40, 40, 40, 2), we randomly generate the problem data as follows:

The computational results of $\mathbf{x}^*$ and $\mathbf{x}$ are shown in Figure 4.

We use the following form of $W$ to solve Example 4.2:

Set $\underline{\gamma} = \overline{\gamma} = 3, \; \mu_0 = 2, \; \mathbf{\mathtt{Maxiter}} = 10^3, \; \nu = 1.2200, \; \sigma = 2.6915, \; \rho = 1.0001 \; \mathrm{and}\; \kappa = 0.711.$ For the given positive integers $m, n$ and $s$, the data are generated by

We run numerical experiments with $(m, n, p, s)$ = (40, 40, 40, 2). Set $\lambda = 10^{-3}$ in (4.2), $\mathbf{x}_0$ = $ones(n, 1)$ and $\varepsilon = 10^{-3}$. From Figures 4 and 5, it can be seen that the output of $\mathbf{x}^k$ is very close to the true solution.

5.   Conclusions

We have intensively studied the composite sparse optimization problem consisting of the sum of a nonsmooth convex function and the ${\mathcal{\ell}_0}$ penalty term of a matrix times the coefficient vector. Considering the original cardinality penalty problem and an exact continuous relaxation problem with capped-${\mathcal{\ell}_1}$ penalty, we have proved several novel and interesting results: the consistency between global minimizers of the relaxation problem and the original problem, and local minimizers of relaxation problems are local minimizers of the original problem. We propose the SGD algorithm based on the smoothing method and the smoothing gradient descent algorithm. Then SGD algorithm has been investigated from both a theoretical and an algorithmic point of view. So we prove that if the sequence generated by the algorithm has an accumulation point, then it is a lifted stationary point of relaxation problem. This well explains why the algorithm is expected to enjoy an appealing performance from the theoretical perspective, which is testified by the numerical experiments. Our initial numerical results confirm the predicted underlying theoretical results.

Author contributions

Wei Yang: Conceptualization, writing-original draft, validation, software; Lili Pan: Conceptualization, supervision, funding acquisition, validation, software; Jinhui Wan: Software, methodology, data curation. All authors have read and approved the final version of the manuscript for publication.

Acknowledgments

The work of the authors was supported by the National Natural Science Foundation of China grants 12271309.

Conflict of interest

The authors declare no conflicts of interest.