![]() |
Let λf(n) be the n-th normalized Fourier coefficient of f, which is a primitive holomorphic cusp form of even integral weight k≥2 for the full modular group SL2(Z). Let also σ(n) and ϕ(n) be the sum-of-divisors function and the Euler totient function, respectively. In this paper, we are able to establish the asymptotic formula of the sum of the hybrid arithmetic function λlf(n)σc(n)ϕd(n) over the sparse sequence {n:n=a2+b2}, namely, ∑n≤xλlf(n)σc(n)ϕd(n)r2(n) for 1≤l≤8, where x is a sufficiently large real number, the function r2(n) denotes the number of representations of n as n=a2+b2, a,b,l∈Z and c,d∈R.
Citation: Huafeng Liu, Rui Liu. The sum of a hybrid arithmetic function over a sparse sequence[J]. AIMS Mathematics, 2024, 9(2): 4830-4843. doi: 10.3934/math.2024234
[1] | Aissa Boukarou, Kaddour Guerbati, Khaled Zennir, Mohammad Alnegga . Gevrey regularity for the generalized Kadomtsev-Petviashvili I (gKP-I) equation. AIMS Mathematics, 2021, 6(9): 10037-10054. doi: 10.3934/math.2021583 |
[2] | Xiaochun Sun, Yulian Wu, Gaoting Xu . Global well-posedness for the 3D rotating Boussinesq equations in variable exponent Fourier-Besov spaces. AIMS Mathematics, 2023, 8(11): 27065-27079. doi: 10.3934/math.20231385 |
[3] | Muhammad Imran Liaqat, Fahim Ud Din, Wedad Albalawi, Kottakkaran Sooppy Nisar, Abdel-Haleem Abdel-Aty . Analysis of stochastic delay differential equations in the framework of conformable fractional derivatives. AIMS Mathematics, 2024, 9(5): 11194-11211. doi: 10.3934/math.2024549 |
[4] | William O. Bray, Ellen Hunter . Wave equations & energy. AIMS Mathematics, 2019, 4(3): 463-481. doi: 10.3934/math.2019.3.463 |
[5] | Phuong Nguyen Duc, Ahmet Ocak Akdemir, Van Tien Nguyen, Anh Tuan Nguyen . Remarks on parabolic equation with the conformable variable derivative in Hilbert scales. AIMS Mathematics, 2022, 7(11): 20020-20042. doi: 10.3934/math.20221095 |
[6] | Kedong Wang, Xianguo Geng, Mingming Chen, Ruomeng Li . Long-time asymptotics for the generalized Sasa-Satsuma equation. AIMS Mathematics, 2020, 5(6): 7413-7437. doi: 10.3934/math.2020475 |
[7] | Said Mesloub, Faten Aldosari . Well posedness for a singular two dimensional fractional initial boundary value problem with Bessel operator involving boundary integral conditions. AIMS Mathematics, 2021, 6(9): 9786-9812. doi: 10.3934/math.2021569 |
[8] | Zhongdi Cen, Jian Huang, Aimin Xu . A posteriori mesh method for a system of singularly perturbed initial value problems. AIMS Mathematics, 2022, 7(9): 16719-16732. doi: 10.3934/math.2022917 |
[9] | Murat A. Sultanov, Vladimir E. Misilov, Makhmud A. Sadybekov . Numerical method for solving the subdiffusion differential equation with nonlocal boundary conditions. AIMS Mathematics, 2024, 9(12): 36385-36404. doi: 10.3934/math.20241726 |
[10] | Dumitru Baleanu, Babak Shiri . Generalized fractional differential equations for past dynamic. AIMS Mathematics, 2022, 7(8): 14394-14418. doi: 10.3934/math.2022793 |
Let λf(n) be the n-th normalized Fourier coefficient of f, which is a primitive holomorphic cusp form of even integral weight k≥2 for the full modular group SL2(Z). Let also σ(n) and ϕ(n) be the sum-of-divisors function and the Euler totient function, respectively. In this paper, we are able to establish the asymptotic formula of the sum of the hybrid arithmetic function λlf(n)σc(n)ϕd(n) over the sparse sequence {n:n=a2+b2}, namely, ∑n≤xλlf(n)σc(n)ϕd(n)r2(n) for 1≤l≤8, where x is a sufficiently large real number, the function r2(n) denotes the number of representations of n as n=a2+b2, a,b,l∈Z and c,d∈R.
Sparse optimization is a core problem of compressed sensing [1,2,3], signal and image processing [4,5,6,7], and high-dimensional statistical learning [4,8], etc. Sparsity is usually characterized by the ℓ0 norm, which is the cardinality of the support set of vector x∈Rn, denoted by ‖x‖0=|supp(x)|=|{i∈{1,2,⋯,n}:xi≠0}|. The penalized formulation of sparse optimization can be expressed as the following cardinality regularized optimization:
minx∈Rnf(x)+λ‖x‖0, | (1.1) |
where f:Rn→R is a loss function depending on the application, λ>0 is the penalty parameter.
Compared with sparse optimization, composite sparse optimization problems enforce certain structural constraints instead of pure sparsity on the coefficients, which arise from many important applications in various fields, such as structural health monitoring [10], fault diagnosis [11], motion planning [12] and impact force identification[13], etc. The most important method is to promote the sparsity of variables through linear transformation [14]. By imposing a regularization matrix W=(W⊤1,…,W⊤p)⊤∈Rp×n on vector x, composite sparse optimization is nicely encapsulated as the following optimization formulation:
minx∈Rnf(x)+λ‖Wx‖0. | (1.2) |
A typical choice of function f in problem (1.2) is the ℓ2 loss function given by f(x)=12‖Ax−b‖22, where A=(A⊤1,…,A⊤m)⊤∈Rm×n and b=(b1,…,bm)⊤∈Rm, and the ℓ1 relaxation of the ℓ0 norm given by ‖Wx‖1, which was first defined and summarized as generalized LASSO with the general formulation of W [15]. Unlike traditional LASSO, solving generalized LASSO efficiently on high- dimensional data is very challenging. A few attempts have been made to improve the efficiency of generalized LASSO, but this requires a specific form of the W to work well [14,15,16], such as the fused LASSO problem [17], the TV regularizer [18] and trending filtering [19].
However, many loss functions of the composite sparse optimization problems cannot be expressed in the form of differentiable functions. The results in [20] showed that the least squares loss function can solve a class of linear regression problems but is not suitable for all types of data. We can choose the outlier that has a strong interference loss function, such as the ℓ1 function, quantile regression function, or more general Huber class function. On the other hand, J. Fan et al. [20] pointed out that using the ℓ1 relaxation often results in a biased estimator, various continuous nonconvex relaxation functions for ℓ0 norm were proposed, such as the smoothly clipped absolute deviation (SCAD) function [20], the hard thresholding function [21], capped-ℓ1 function [22,23,24,25,26], the transformed ℓ1 function [27], etc. Here, we are interested in the capped-ℓ1 function as the relaxation function of the ℓ0 norm, which is a simple relaxation function that satisfies specific properties. Z. Shen et al. [40] applied locally Lipschitz continuous scaled folded concave functions to the approximate ℓ0 pseudo-norm. A generic nonsmooth but convex framework was established to gradually approximate the scaled folded concave functions. Numerical experimental results showed the proposed framework and algorithms admitted the exact sparsity-induced capability of the ℓ0 pseudo-norm. Q. Chen et al.[41] first explored using a class of locally Lipschitz scale folded concave functions to approach the ℓ0. Then, a convex half-absolute method was proposed to precisely approximate these nonconvex nonsmooth functions. A double iterative algorithm was considered to solve the convex-relaxed composite optimization problems. Both [40] and [41] established a generic nonsmooth convex framework that gradually approximates these scale-folded concave functions based on the Legendre-Fenchel transformation to avoid directly solving a nonconvex optimization problem. However, we use a smoothing function for approximation to achieve the goal of solving the nonconvex optimization problem. The advantages of capped-ℓ1 function have been explored in various fields. For example, the authors in [28] put forward a capped ℓ1-norm Sparse Representation method (CSR) for graph clustering. The proposed model learned the optimal graph for clustering by integrating sparse representation and capped ℓ1-norm loss function. In order to utilize the advantages of the twin extreme learning machine and FDA, Z. Xue et al. [29] first put forward a novel classifier named Fisher-regularized twin extreme learning machine (FTELM). Also considering the instability of the ℓ2-norm for the outliers, authors introduced the capped ℓ1-norm into the FTELM model and proposed a more robust capped ℓ1-norm FTELM (Cℓ1-FTELM) model. The capped ℓ1 function was also discussed in the context of sparse group ℓ0 regularized algorithms by [30]. It's worth noting that reference [9] gave an exact continuous relaxation problem with capped-ℓ1 penalty for nonsmooth convex loss function with cardinality penalty in the sense that both problems have the same optimal solution set. Moreover, a smoothing proximal gradient algorithm for finding a lifted stationary point of the continuous relaxation model was proposed. Regarding the solution of relaxation problems, T. Zhang [42] presented a multi-stage convex relaxation scheme for solving problems with non-convex objective functions. However, only parameter estimation performance was analyzed in [42]. Unfortunately, the result in [42] does not directly imply that multi-stage convex relaxation achieves unbiased recovery of the support set. H. Zhou et al. [43] proposed a new unified algorithm based on the local linear approximation (LLA) for maximizing the penalized likelihood for a broad class of concave penalty functions. It did not eliminate the bias issue. Here, we extend the results in [9] to composite sparse optimization and give a smoothing gradient descent algorithm for the continuous relaxation problem. The new algorithm exploits the piecewise linearity of the capped-ℓ1 penalty term in the relaxation problem. In view of the composite sparsity, if the subproblem in the algorithm does not have a closed solution, then our relaxation problem model analogizes the ℓ1 penalty model for solving LASSO problems, using the smoothing gradient method to solve the subproblem. We prove that if the sequence generated by the algorithm has an accumulation point, then it is a lifted stationary point of relaxation problem.
In this paper, we consider the following composite sparse regression problem with cardinality penalty,
minx∈ΩWℓ0(x):=f(x)+λ‖Wx‖0, | (1.3) |
where f:Rn→R is a convex (not necessarily smooth) and bounded from below function, λ is a positive parameter, and Ω={x∈Rn:l≤Wx≤u}. For example, the ℓ1 loss function given by
f(x)=1m‖Ax−b‖1, | (1.4) |
or the censored regression problem with
f(x)=1mm∑i=1|max{Aix,0}−bi|. | (1.5) |
For a given parameter ν>0, the continuous relaxation of the ℓ0 penalty with the capped-ℓ1 function ϕ is given by
ϕ(t)=min{1,|t|ν}. | (1.6) |
We consider the following continuous optimization problem to solve (1.3):
minx∈ΩW(x):=f(x)+λΦ(Wx), | (1.7) |
where Φ(Wx)=∑pi=1ϕ(Wix).
Composite sparse optimization has attracted much attention recently. In [15], a dual path algorithm was proposed for the generalized LASSO problem with any formulation of W. If the composite optimization problem is convex and the W is the general linear map, one feasible approach is to apply the alternating direction method of multipliers (ADMM)[31]. In [31], the author proposed a dual method for the variational problem in the form of inf{f(Av)+g(v)}. Z. J. Bai [32] aimed to provide a coordinate gradient descent method with stepsize chosen by an Armijo-type rule to solve the problem minxf(x)+c‖Lx‖1 and minx‖Ax−b‖22+c‖Lx‖1 efficiently, especially when the problems dimension is large.
In this paper, we use the exact continuous relaxation problem with capped-ℓ1 function to solve optimization problem (1.3) and present a smoothing gradient descent (SGD) algorithm. Since W is a general linear mapping and Φ(Wx) is an inseparable function, which makes the proximal gradient algorithm unable to be explicitly applied, we approximately solve the subproblem in the SGD algorithm. We prove that if there is an accumulation point, then the accumulation point is a lifted stationary point of (1.7).
Notation. We denote N={0,1,…}. For x∈Rn and δ>0, let ‖x‖:=‖x‖2 and Bδ(x) means the open ball centered at x with radius δ. For a nonempty, closed, and convex set Ω⊆Rn, NΩ(x) means the normal cone to Ω at x∈Ω. σmin(W) is the minimum singular value of W. We denote ‖x‖∞=max{|x1|,|x2|,…,|xn|}.
Before starting this section, we first make the following two assumptions:
Assumption 2.1. Function f is Lipschitz continuous on Ω with Lipschitz constant Lf>0 and matrix W has full column rank.
Assumption 2.2. Parameter ν in (1.6) satisfies 0<ν<¯ν:=λσmin(W)Lf.
We suppose Assumptions 2.1 and 2.2 hold throughout the paper and assume that Lf is large enough such that Lf≥λσmin(W)Γ, where
Γ:=min{|li|,ui:li≠0,ui≠0,i=1,…,p}. |
When f is defined by the ℓ1 loss function (1.4) or the censored regression loss function (1.5), Lf can be taken as Lf=‖A‖∞.
We first give the definition of lifted stationary points of (1.7) as that in [33]. Since function ϕ in (1.6) can be rephrased as
ϕ(t)=1ν|t|−max{θ1(t),θ2(t),θ3(t)} |
with θ1(t)=0, θ2(t)=tν−1 and θ3(t)=−tν−1 for t∈R, we denote
D(t):={i∈{1,2,3}:θi(t)=max{θ1(t),θ2(t),θ3(t)}} | (2.1) |
and
Dp(Wx):=Πpi=1D(Wix). | (2.2) |
Definition 2.1. Point x∈Ω is called a lifted stationary point of (1.7) if there exists d=(d1,…,dp)⊤∈Dp(Wx) such that
λp∑i=1(θ′di(Wix))⊤∈∂f(x)+λνp∑i=1W⊤iϑi(x)+NΩ(x), | (2.3) |
where
ϑi(x){=1ifWix>0,∈[−1,1]if|Wix|=0,=−1ifWix<0. | (2.4) |
Under the definition of the range of ν in Assumption 2.2, we first prove that the element in Dp(Wx) for a lifted stationary point satisfying (2.3) is unique.
Proposition 2.2. If ˉx is a lifted stationary point of (1.7), then the vector dWˉx=(dWˉx1,…,dWˉxp)⊤∈Dp(Wˉx) satisfying (2.3) is unique. In particular, for i=1,…,p,
dWˉxi={1if|Wiˉx|<ν,2ifWiˉx≥ν,3ifWiˉx≤−ν. | (2.5) |
Proof. If |Wiˉx|≠ν, the statement is clearly valid. Hence, we only need to consider the case |Wiˉx|=ν. When Wiˉx=ν, since D(Wiˉx)={1,2}, arguing by contradiction, we assume (2.3) holds with dWˉxi=1, so ϑi(ˉx)=1. By ν<ˉν, we have Wiˉx∈(li,ui), and by (2.3), there exists ξ(ˉx)∈∂f(ˉx) such that
0=ξ(ˉx)+λν∑i:|Wiˉx|≤νW⊤iϑi(ˉx), | (2.6) |
where ϑi(ˉx) is defined as (2.4).
It is easy to observe that the following relation holds
‖∑i:|Wiˉx|≤νW⊤iϑi(ˉx)‖2≥σmin(W). | (2.7) |
In fact, from the definition of the minimum singular value of W,
σmin(W)=min{‖Wx‖2‖x‖2:x≠0}=min{‖Wx‖2:‖x‖2=1}, |
we have
σmin(W)=min{‖Wx‖2‖x‖2:x≠0}≤min{‖Wx‖2‖x‖2:‖x‖2≥1,‖x‖∞≤1}≤min{‖Wx‖2:‖x‖2≥1,‖x‖∞≤1}≤min{‖Wx‖2:‖x‖2=1}=σmin(W). |
Then, we see that
σmin(W)=min{‖Wx‖2:‖x‖2≥1,‖x‖∞≤1}. |
From the definition of ϑi(ˉx) (2.4), this yields that
‖∑i:|Wiˉx|≤νW⊤iϑi(ˉx)‖2≥σmin(WI)≥σmin(W), |
where WI is the submatrix consisting of the rows in W indexed by I:={i:|Wiˉx|≤ν} [34].
Combining (2.6) and (2.7), we have
λσmin(W)ν≤λν‖∑i:|Wiˉx|≤νW⊤iϑi(ˉx)‖=‖ξ(ˉx)‖≤Lf. |
This leads to a contradiction to ν<λσmin(W)Lf. Then, (2.5) holds for Wiˉx=ν. Similar analysis can be given for the case that Wiˉx=−ν, which completes the proof.
For a given d=(d1,…,dp)⊤∈Dp:={d∈Rp:di∈{1,2,3},i=1,⋯,p}, we define
Φd(Wx):=p∑i=1|Wix|ν−p∑i=1θdi(Wix), | (2.8) |
which is convex with respect to x. It can be verified that
Φ(Wx)=mind∈DpΦd(Wx),∀x∈Ω. |
In particular, for a fixed ˉx∈Ω, Φ(Wˉx)=ΦdWˉx(Wˉx) and the following relation holds
ˉxisaliftedstationarypointof(1.7)⇔ˉx∈argminx∈Ωf(x)+λΦdWˉx(Wx). | (2.9) |
Next lemma describes a lower bound property.
Lemma 2.3. If ˉx∈Ω is a lifted stationary point of (1.7), then it holds that
Wiˉx∈(−ν,ν)⇒Wiˉx=0,∀i=1,…,p. | (2.10) |
Proof. Suppose that ˉx is a lifted stationary point of (1.7). Now we assume that Wiˉx∈(−ν,ν)∖{0} for some i∈1,…,p. So from (2.5) and Assumption 2.1, we have dWˉxi=1 and Wiˉx∈(li,ui). By (2.3), there exists ξ(ˉx)∈∂f(ˉx). We have
0=ξ(ˉx)+λν∑i:|Wiˉx|<νW⊤iϑi(ˉx). |
Through the analysis in the proof of Proposition 2.2, combining (2.6) and (2.7), we have
λσmin(W)ν≤λν‖∑i:|Wiˉx|<νW⊤iϑi(ˉx)‖=‖ξ(ˉx)‖≤Lf, |
which leads to a contradiction to ν<λσmin(W)Lf. Consequently, Wiˉx∈(−ν,ν) implies Wiˉx=0 for i∈1,…,p {and} the proof is completed.
This subsection discusses the relationship between the global minimizers and local minimizers of (1.3) and (1.7). First, Theorem 2.4 discusses the relationship between the local minimizers of (1.3) and (1.7). Second, Theorem 2.5 states that (1.3) and (1.7) have the same global minimizers. We use the lower bound property mentioned in Lemma 2.3 to prove Theorems 2.4 and 2.5.
Theorem 2.4. If ˉx is a lifted stationary point of (1.7), then it is a local minimizer of (1.3) and the objective functions have the same value at ˉx, i.e., f(ˉx)+λΦ(Wˉx)=f(ˉx)+λ‖Wˉx‖0.
Proof. Combining the lower bound property of Wˉx in (2.10) and the definition of ΦdWˉx defined in (2.8), for any x∈Rn, we have
ΦdWˉx(Wx):=p∑i=1|Wix|ν−p∑i=1θdWˉxi(Wix)=∑i:|Wiˉx|≥ν1+∑i:|Wiˉx|<ν|Wix|ν=‖Wˉx‖0+∑i:Wiˉx=0|Wix|ν. |
Then
ΦdWˉx(Wx)≤‖Wx‖0,∀x∈Bϱ(ˉx),ϱ>0. | (2.11) |
Combining this with Φ(Wˉx)=‖Wˉx‖0 and (2.9), we have
f(ˉx)+λ‖Wˉx‖0≤f(x)+λ‖Wx‖0,∀x∈Ω∩Bϱ(ˉx). |
Thus, ˉx is a local minimizer of (1.7).
Theorem 2.4 indicates that any lifted stationary point of (1.7) is a local minimizer of (1.3), which means that any local minimizer of (1.7) is also certainly a local minimizer of (1.3).
Theorem 2.5. If ˉx∈Ω is a global minimizer of (1.3) if and only if it is a global minimizer of (1.7). Moreover, problems (1.3) and (1.7) have the same optimal value.
Proof. On the one hand, let ˉx∈Ω be a global minimizer of (1.7), and according to Definition 2.1, then we can obtain that ˉx is a lifted stationary point of (1.7). By (2.10), from Wiˉx∈(−ν,ν), then Wiˉx=0, so it gives Φ(Wˉx)=‖Wˉx‖0. We have
f(ˉx)+λ‖Wˉx‖0=f(ˉx)+λΦ(Wˉx)≤ f(x)+λΦ(Wx)≤f(x)+λ‖Wx‖0,∀x∈Ω, |
where the last inequality uses Φ(Wx)≤‖Wx‖0. Therefore, ˉx is a global minimizer of (1.3).
On the other hand, let ˉx∈Ω be a global minimizer of (1.3). Assume on the contrary ˉx is not a solution of (1.7). Let ˆx be a global minimizer of (1.7), we obtain
f(ˆx)+λΦ(Wˆx)<f(ˉx)+λΦ(Wˉx). |
From
Φ(Wˆx)=‖Wˆx‖0andΦ(Wˉx)≤‖Wˉx‖0, |
we have
f(ˆx)+λ‖Wˆx‖0<f(ˉx)+λ‖Wˉx‖0. |
This contradicts the global optimality of ˉx for (1.3). Hence ˉx is a global minimizer of (1.7). Therefore, (1.3) and (1.7) have the same global minimizers and optimal values.
When f is convex, ˉx is a local minimizer of (1.3) if and only if ˉx∈Ω satisfies
0∈∂f(ˉx)+NΩ(ˉx). | (2.12) |
which is often used as a criterion for the local minimizers of problem (1.3).
Definition 2.6. We call ˉx∈Ω a ν-strong local minimizer of (1.3), if there exists ¯ξ∈∂f(ˉx) and ¯η∈NΩ(ˉx) such that for any i∈supp(Wˉx), it holds
0=¯ξ+¯ηand|Wiˉx|≥ν. |
By (2.12), any ν-strong local minimizer of (1.3) is a local minimizer of it. Below we provide a result on the relationship between the ν-strong local minimizers of (1.3) and the lifted stationary points of (1.7).
Proposition 2.7. ˉx∈Ω is a ν-strong local minimizer of (1.3) if and only if it is a lifted stationary point of (1.7).
Proof. First, by (2.9), we see that if ˉx is a lifted stationary point of (1.7), then
Wℓ0(ˉx)=f(ˉx)+λ‖Wˉx‖0=f(ˉx)+λΦ(Wˉx)=f(ˉx)+λΦdWˉx(Wˉx)≤f(x)+λΦdWˉx(Wx),∀x∈Ω. |
Combining the Lemma 2.3 and ΦdWˉx(Wx)≤‖Wx‖0,∀x∈Bϱ(ˉx),ϱ>0 in (2.11), then we have
Wℓ0(ˉx)≤Wℓ0(x),∀x∈Ω∩Bϱ(ˉx), |
so ˉx is a ν-strong local minimizer of (1.3).
Next, because ˉx is a ν-strong local minimizer of (1.3), it is also a local minimizer of (1.3), suppose ˉx is a local minimizer of (1.3) but not a local minimizer of (1.7). Then there exists a local minimizer of (1.7) denoted by ˆx, combining (2.9), ΦdWˆx(Wˉx)≤‖Wˉx‖0,∀ˉx∈Bϱ(ˆx) in (2.11) and Φ(Wˆx)=‖Wˆx‖0, we have
f(ˆx)+λ‖Wˆx‖0=f(ˆx)+λΦ(Wˆx)=f(ˆx)+λΦdWˆx(Wˆx)≤f(ˉx)+λΦdWˆx(Wˉx)≤f(ˉx)+λ‖Wˉx‖0,∀ˉx∈Bϱ(ˆx), |
which leads to a contradiction. Thus, the local minimizer of (1.3) is the local minimizer of (1.7), that is to say, the ν-strong local minimizer of (1.3) is the lifted stationary point of (1.7).
We use Table 1 to clearly demonstrate the relationship between (1.3) and (1.7).
![]() |
The main content of this section is to find the lifted stationary point of (1.7). Due to the existence of matrix W, we cannot express the explicit solution using the proximal gradient method. We first approximate f by a smoothing function and propose some preliminary theories on the smoothing methods; the second section proposes our algorithm; and the third section conducts a convergence analysis on the proposed algorithm for solving (1.7).
Throughout this paper, we approximate the loss function f by a smoothing function ˜f in (1.7). When it is clear from the context, the derivative of ˜f(x,μ) with respect to x is simply denoted as ∇˜f(x,μ). We denote
˜Wd(x,μ):=˜f(x,μ)+λΦd(Wx),˜W(x,μ):=˜f(x,μ)+λΦ(Wx), |
where smoothing parameter μ>0 and d∈Dp. For any fixed μ>0 and d∈Dp, ˜Wd(x,μ) is nonsmooth and convex, and ˜W(x,μ) is nonsmooth and nonconvex. Due to
Φ(Wx)=mind∈DpΦd(Wx),∀x∈Ω, |
we obtain
˜Wd(x,μ)≥˜W(x,μ),∀d∈Dp,x∈Ω,μ∈(0,ˉμ]. | (3.1) |
The following definition describes some theories about the smoothing function ˜f, which is frequently used in the proof of convergence analysis.
Definition 3.1 We call ˜f:Rn×[0,ˉμ]→R with ˉμ>0 a smoothing function of the convex function f in (1.7), if ˜f(⋅,μ) is continuously differentiable in Rn for any fixed μ>0 and satisfies the following conditions:
(ⅰ) limz→x,μ↓0˜f(z,μ)=f(x),∀x∈Ω;
(ⅱ) (convexity) ˜f(x,μ) is convex with respect to x in Ω for any fixed μ>0;
(ⅲ) (gradient consistency) {limz→x,μ↓0▽z˜f(z,μ)}⊆∂f(x),∀x∈Ω;
(ⅳ) (Lipschitz continuity with respect to μ) there exists a positive constant κ such that
|˜f(x,μ2)−˜f(x,μ1)|≤κ|μ1−μ2|,∀x∈Ω,μ1,μ2∈[0,ˉμ]; |
(ⅴ) (Lipschitz continuity with respect to x) there exists a constant L>0 such that for any μ∈(0,ˉμ],▽x˜f(⋅,μ) is Lipschitz continuous on Ω with Lipschitz constant Lμ−1.
By virtue of Definition 3.1-(ⅳ), we obtain that
|˜f(x,μ)−f(x)|≤κμ,∀x∈Ω,0<μ≤ˉμ. | (3.2) |
Next, we aim to solve the following problem with μ>0 and vector d∈Dp
minx∈Ω˜Wd(x,μ)=˜f(x,μ)+λΦd(Wx), | (3.3) |
by introducing an approximation of ˜Wd(⋅,μ) around a given point z as follows:
Gd,γ(x,z,μ)=˜f(z,μ)+⟨x−z,∇˜f(z,μ)⟩+12γμ−1‖x−z‖2+λΦd(Wx) | (3.4) |
with a constant γ>0. Φd(Wx) is convex with respect to x for any fixed d∈Dp, function Gd,γ(x,z,μ) is a strongly convex function with respect to x for any fixed d,γ,z and μ. Then, we solve the following problem
minx∈ΩGd,γ(x,z,μ) |
to find the approximate solution of (3.3).
In this subsection, we propose a new algorithm (see Algorithm 1) for finding a lifted stationary point of (1.7). Specially, since W is a general linear mapping and Φ(Wx) is an inseparable function, which makes the smoothing proximal gradient algorithm [9] cannot explicitly solve a subproblem. The proposed algorithm combines the smoothing method and the smoothing gradient descent algorithm, so we call it the smoothing gradient descent (SGD) algorithm. We use the SGD algorithm to obtain approximate solutions of the subproblem. Let
Ps={k∈N:μk+1≠μk}, |
and denote psr the rth smallest number in Ps. Then, we can obtain the following updating method of {μk}
μk=μpsr+1=μ0(psr+1)σ,∀psr+1≤k≤psr+1, | (3.5) |
which will be used in the proof of Lemmas 3.2 and 3.4.
Algorithm 1: Smoothing Gradient Descent (SGD) algorithm |
Require: Take x−1=x0∈Ω and μ−1=μ0∈(0,ˉμ]. Give parameters ρ>1, σ>12, α>0 and 0<γ_<ˉγ. Set k=0. |
1: while a termination criterion is not met do |
2: Step 1.Choose γk∈[γ_,ˉγ] and let dk≜, where d^{W\mathbf{x}^k} is defined in (2.5). |
3: Step 2. |
3: 2a) Compute |
4: |
{\hat{\mathbf{x}}}^{k+1} = \arg\min\limits_{\mathbf{x}\in\Omega}\; G_{d^k, \gamma^k}(\mathbf{x}, \mathbf{x}^{k}, \mu_k). (3.6) |
4: 2b) If {\hat{\mathbf{x}}}^{k+1} satisfies |
5: |
\widetilde{\mathcal{W}}^{d^k}({\hat{\mathbf{x}}}^{k+1}, \mu_k)\leq G_{d^k, \gamma^k}({\hat{\mathbf{x}}}^{k+1}, \mathbf{x}^{k}, \mu_k). (3.7) |
6: Set |
7: |
\mathbf{x}^{k+1} = {\hat{\mathbf{x}}}^{k+1} (3.8) |
8: and go to Step 3. Otherwise, let \gamma_k = \rho\gamma_k and return to 2a). |
9: Step 3. If |
10: |
\widetilde{\mathcal{W}}(\mathbf{x}^{k+1}, \mu_k)+\kappa\mu_k-\widetilde{\mathcal{W}}(\mathbf{x}^{k}, \mu_{k-1})-\kappa\mu_{k-1}\leq-\alpha\mu_k^2, (3.9) |
11: set \mu_{k+1} = \mu_k , otherwise, set |
12: |
\mu_{k+1} = \frac{\mu_0}{(k+1)^\sigma}. (3.10) |
13: Increment k by one and return to Step 1. |
14: end while |
Lemma 3.2. The proposed SGD algorithm is well-defined, and the sequences \{\mathbf{x}^k\} , \{\gamma^k\} and \{\mu_k\} generated by it own the following properties:
(ⅰ) \{\mathbf{x}^k\}\subseteq\Omega and \{\gamma_k\}\subseteq[\underline{\gamma}, \max\{\overline{\gamma}, \rho L\}];
(ⅱ) there are infinite elements in \mathcal{P}^s and \lim_{k\rightarrow \infty}\mu_k = 0.
Proof. (ⅰ) By organizing (3.7), we can obtain
\tilde{f}(\hat{\mathbf{x}}^{k+1},\mu_k)\leq\tilde{f}({\mathbf{x}}^{k},\mu_k)+\langle\nabla\tilde{f}({\mathbf{x}}^{k},\mu_k),\hat{\mathbf{x}}^{k+1}-\mathbf{x}^k\rangle+\frac{1}{2}\gamma_k\mu_k^{-1}\|\hat{\mathbf{x}}^{k+1}-\mathbf{x}^k\|^2. |
According to Definition 3.1-(ⅴ), (3.7) holds when \gamma_k\geq L . Thus the updating of \gamma_k in Step 2 is at most \log_\rho(\frac{L}{\underline{\gamma}})+1 times at each iteration. Hence, the SGD algorithm is well-defined, and we have that \gamma_k\leq\max\{\overline{\gamma}, \rho L\}, \forall k\in\mathbb{N}. From (3.8), it is easy to verify that \mathbf{x}^{k+1}\in\Omega by \mathbf{x}^{k}\in\Omega and \hat{\mathbf{x}}^{k+1}\in\Omega .
(ⅱ) Since \{\mu_k\} is non-increasing, to prove (ⅱ), we assume that \lim_{k\rightarrow \infty}\mu_k = \hat{\mu} > 0 by contradiction. If \{\mu_k\} converges to a non-zero value, then the iteration of (3.10) is finite, which means that there exists \mathrm{K}\in\mathbb{N} such that \mu_k = \hat{\mu}, \forall k\geq\mathrm{K} . Substituting \hat{\mu} into (3.9), we obtain
\widetilde{\mathcal{W}}(\mathbf{x}^{k+1},\mu_k)+\kappa\mu_k-\widetilde{\mathcal{W}}(\mathbf{x}^{k},\mu_{k-1})-\kappa\mu_{k-1}\leq-\alpha\hat{\mu}^2, \forall k\geq\mathrm{K}+1. |
By the above inequality, we have
\begin{equation} \begin{aligned} \lim\limits_{k\rightarrow \infty}\widetilde{\mathcal{W}}(\mathbf{x}^{k+1},\mu_k)+\kappa\mu_k = -\infty. \end{aligned} \end{equation} | (3.11) |
However, by \{\mathbf{x}^k\}\subseteq\Omega , (3.2) and Theorem 2.5, then
\begin{equation} \begin{aligned} \widetilde{\mathcal{W}}(\mathbf{x}^{k+1},\mu_k)+\kappa\mu_k\geq\mathcal{W}(\mathbf{x}^{k+1})\geq\min\limits_{\mathbf{x}\in\Omega}\mathcal{W}(\mathbf{x}) = \min\limits_{\mathbf{x}\in\Omega}\mathcal{W}_{\ell_0}(\mathbf{x}),\; \; \forall k\geq\mathrm{K}, \end{aligned} \end{equation} | (3.12) |
(3.11) and (3.12) are contradictory; (ⅱ) holds.
Lemma 3.3. For any k\in\mathbb{N} , we have
\begin{equation} \begin{aligned} \widetilde{\mathcal{W}}(\mathbf{x}^{k+1},\mu_k)-\widetilde{\mathcal{W}}(\mathbf{x}^{k},\mu_k)\leq-\frac{1}{2}\gamma_k\mu_k^{-1}\|\mathbf{x}^{k+1}-\mathbf{x}^k\|^2, \end{aligned} \end{equation} | (3.13) |
which implies \{\widetilde{\mathcal{W}}(\mathbf{x}^{k+1}, \mu_k)+\kappa\mu_k\} is non-increasing and \lim\limits_{k\rightarrow \infty}\widetilde{\mathcal{W}}(\mathbf{x}^{k+1}, \mu_k) = \lim\limits_{k\rightarrow \infty}\mathcal{W}(\mathbf{x}^{k}).
Proof. Since G_{d^k, \gamma_k}(\mathbf{x}, \mathbf{x}^k, \mu_k) is strongly convex with modulus \gamma_k\mu_k^{-1}, we have
\begin{equation} \begin{aligned} G_{d^k,\gamma_k}(\mathbf{x},\mathbf{x}^k,\mu_k)\geq G_{d^k,\gamma_k}(\hat{\mathbf{x}}^{k+1},\mathbf{x}^k,\mu_k)+\langle\nabla G_{d^k,\gamma_k}(\hat{\mathbf{x}}^{k+1},\mathbf{x}^k,\mu_k),\mathbf{x}-\hat{\mathbf{x}}^{k+1}\rangle\\ +\frac{1}{2}\gamma_k\mu_k^{-1}\|\hat{\mathbf{x}}^{k+1}-\mathbf{x}\|^2, \end{aligned} \end{equation} | (3.14) |
using the definition of \hat{\mathbf{x}}^{k+1} in (3.6) and \mathbf{x}^{k+1} = \hat{\mathbf{x}}^{k+1} when (3.7) holds, we obtain
G_{d^k,\gamma_k}(\mathbf{x}^{k+1},\mathbf{x}^{k},\mu_k)\leq G_{d^k,\gamma_k}(\mathbf{x},\mathbf{x}^k,\mu_k)-\frac{1}{2}\gamma_k\mu_k^{-1}\|\mathbf{x}^{k+1}-\mathbf{x}\|^2,\forall \mathbf{x}\in\Omega. |
By the definition of function G_{d^k, \gamma_k} given in (3.4), organizing the inequalities above, we have
\begin{equation} \begin{aligned} \lambda\Phi^{d^k}(W\mathbf{x}^{k+1})\leq&\lambda\Phi^{d^k}(W\mathbf{x})+\langle \mathbf{x}-\mathbf{x}^{k+1},\nabla\tilde{f}(\mathbf{x}^k,\mu_k)\rangle\\ &+\frac{1}{2}\gamma_k\mu_k^{-1}\|\mathbf{x}-\mathbf{x}^k\|^2 -\frac{1}{2}\gamma_k\mu_k^{-1}\|\mathbf{x}^{k+1}-\mathbf{x}^k\|^2\\ &-\frac{1}{2}\gamma_k\mu_k^{-1}\|\mathbf{x}^{k+1}-\mathbf{x}\|^2. \end{aligned} \end{equation} | (3.15) |
Moreover, (3.7) can be written as
\begin{equation} \begin{aligned} \widetilde{\mathcal{W}}^{d^k}(\mathbf{x}^{k+1},\mu_k)\leq&\tilde{f}(\mathbf{x}^k,\mu_k)+\langle \mathbf{x}^{k+1}-\mathbf{x}^k,\nabla\tilde{f}(\mathbf{x}^k,\mu_k)\rangle\\ &+\frac{1}{2}\gamma_k\mu_k^{-1}\|\mathbf{x}^{k+1}-\mathbf{x}^k\|^2 +\lambda\Phi^{d^k}(W\mathbf{x}^{k+1}). \end{aligned} \end{equation} | (3.16) |
Summing up (3.15) and (3.16), we obtain that
\begin{equation} \begin{aligned} \widetilde{\mathcal{W}}^{d^k}(\mathbf{x}^{k+1},\mu_k)\leq&\tilde{f}(\mathbf{x}^k,\mu_k)+\lambda\Phi^{d^k}(W\mathbf{x})+\langle \mathbf{x}-\mathbf{x}^k,\nabla\tilde{f}(\mathbf{x}^k,\mu_k)\rangle\\ &+\frac{1}{2}\gamma_k\mu_k^{-1}\|\mathbf{x}-\mathbf{x}^k\|^2-\frac{1}{2}\gamma_k\mu_k^{-1}\|\mathbf{x}^{k+1}-\mathbf{x}\|^2,\forall \mathbf{x}\in\Omega. \end{aligned} \end{equation} | (3.17) |
For a fixed \mu > 0 , the convexity of \tilde{f}(\mathbf{x}, \mu) with respect to \mathbf{x} indicates
\begin{equation} \begin{aligned} \tilde{f}(\mathbf{x}^k,\mu_k)+\langle \mathbf{x}-\mathbf{x}^k,\nabla\tilde{f}(\mathbf{x}^k,\mu_k)\rangle\leq\tilde{f}(\mathbf{x},\mu_k),\forall \mathbf{x}\in\Omega. \end{aligned} \end{equation} | (3.18) |
Combining (3.17) and (3.18) and utilizing the definition of \widetilde{\mathcal{W}}^{d^k} , one has
\begin{equation} \begin{aligned} \widetilde{\mathcal{W}}^{d^k}(\mathbf{x}^{k+1},\mu_k)\leq\widetilde{\mathcal{W}}^{d^k}(\mathbf{x},\mu_k)+\frac{1}{2}\gamma_k\mu_k^{-1}\|\mathbf{x}-\mathbf{x}^k\|^2-\frac{1}{2}\gamma_k\mu_k^{-1}\|\mathbf{x}^{k+1}-\mathbf{x}\|^2,\\ \forall \mathbf{x}\in\Omega. \end{aligned} \end{equation} | (3.19) |
Letting \mathbf{x} = \mathbf{x}^k in (3.19) and by d^k = d^{{W\mathbf{x}}^k} , we obtain
\begin{equation} \begin{aligned} \widetilde{\mathcal{W}}^{d^k}(\mathbf{x}^{k+1},\mu_k)+\frac{1}{2}\gamma_k\mu_k^{-1}\|\mathbf{x}^{k+1}-\mathbf{x}^k\|^2\leq\widetilde{\mathcal{W}}(\mathbf{x}^{k},\mu_k). \end{aligned} \end{equation} | (3.20) |
Because \widetilde{\mathcal{W}}^{d^k}(\mathbf{x}^{k+1}, \mu_k)\geq\widetilde{\mathcal{W}}(\mathbf{x}^{k+1}, \mu_k), (3.20) leads to (3.13). Due to Definition 3.1-(iv), we have
\tilde{f}(\mathbf{x}^k,\mu_{k-1})\geq\tilde{f}(\mathbf{x}^k,\mu_k)-\kappa(\mu_{k-1}-\mu_{k}), |
then, it follows that
\widetilde{\mathcal{W}}(\mathbf{x}^{k},\mu_k)\leq\widetilde{\mathcal{W}}(\mathbf{x}^{k},\mu_{k-1})+\kappa(\mu_{k-1}-\mu_k), |
by (3.13), we obtain
\begin{equation} \begin{aligned} \widetilde{\mathcal{W}}(\mathbf{x}^{k+1},\mu_k)+\kappa\mu_k+\frac{1}{2}\gamma_k\mu_k^{-1}\|\mathbf{x}^{k+1}-\mathbf{x}^k\|^2\leq\widetilde{\mathcal{W}}(\mathbf{x}^{k},\mu_{k-1})+\kappa\mu_{k-1}, \end{aligned} \end{equation} | (3.21) |
(3.21) implies the non-increasing property of \{\widetilde{\mathcal{W}}(\mathbf{x}^{k+1}, \mu_k)+\kappa\mu_k\}. This result and (3.12) ensure the existence of \lim_{k\rightarrow \infty}\widetilde{\mathcal{W}}(\mathbf{x}^{k+1}, \mu_k)+\kappa\mu_k . By virtue of \lim_{k\rightarrow \infty}\mu_k = 0 and Definition 3.1-(i), we obtain
\lim\limits_{k\rightarrow \infty}\widetilde{\mathcal{W}}(\mathbf{x}^{k+1},\mu_k) = \lim\limits_{k\rightarrow \infty}\mathcal{W}(\mathbf{x}^k). |
The proof is completed.
Lemma 3.4. The following statements hold:
(ⅰ) \sum_{k = 0}^\infty\gamma_k\mu_k^{-1}\|\mathbf{x}^{k+1}-\mathbf{x}^k\|^2\leq2(\mathcal{W}(\mathbf{x}^0, \mu_{-1})+\kappa\mu_{-1}-\min_\Omega\mathcal{W});
(ⅱ) \sum_{k = 0}^\infty\mu_k^2\leq\Lambda with \Lambda = \frac{1}{\alpha}(\widetilde{\mathcal{W}}(\mathbf{x}^0, \mu_{-1})+\kappa\mu_{-1}-\min_{\mathbf{x}\in\Omega}\mathcal{W}(\mathbf{x}))+\frac{2\mu_0^2\sigma}{2\sigma-1} < \infty ;
Proof. (ⅰ) Recalling (3.21), for all k\in\mathbb{N} , we obtain
\begin{equation} \begin{aligned} \gamma_k\mu_k^{-1}\|\mathbf{x}^{k+1}-\mathbf{x}^k\|^2\leq2(\widetilde{\mathcal{W}}(\mathbf{x}^k,\mu_{k-1})+\kappa\mu_{k-1}-\widetilde{\mathcal{W}}(\mathbf{x}^{k+1},\mu_{k})-\kappa\mu_{k}). \end{aligned} \end{equation} | (3.22) |
Now adding up the above inequality over k = 0, \ldots, \mathrm{K} , it gives
\begin{equation} \begin{aligned} \sum\limits_{k = 0}^\mathrm{K}\gamma_k\mu_k^{-1}\|\mathbf{x}^{k+1}-\mathbf{x}^k\|^2\leq2(\widetilde{\mathcal{W}}(\mathbf{x}^0,\mu_{-1})+\kappa\mu_{-1}-\widetilde{\mathcal{W}}(\mathbf{x}^{\mathrm{K}+1},\mu_{\mathrm{K}})-\kappa\mu_{\mathrm{K}}). \end{aligned} \end{equation} | (3.23) |
By letting \mathrm{K} in (3.22) tend to infinity and along with (3.12), we obtain (ⅰ).
(ⅱ) From (3.5), we have
\begin{equation} \begin{aligned} \sum\limits_{k\in\mathcal{P}^s}\mu_k^2 = \sum\limits_{r = 1}^\infty\frac{\mu_0^2}{(p_r^s+1)^{2\sigma}}\leq\sum\limits_{k = 1}^\infty\frac{\mu_0^2}{k^{2\sigma}}\leq\frac{2\mu_0^2\sigma}{2\sigma-1}, \end{aligned} \end{equation} | (3.24) |
where p_r^s is the r th smallest element in \mathcal{P}^s . When k\notin\mathcal{P}^s , (3.9) gives
\alpha\mu_k^2\leq\widetilde{\mathcal{W}}(\mathbf{x}^k,\mu_{k-1})+\kappa\mu_{k-1}-\widetilde{\mathcal{W}}(\mathbf{x}^{k+1},\mu_{k})-\kappa\mu_{k}, |
which together with the non-increasing property of \{\widetilde{\mathcal{W}}(\mathbf{x}^{k+1}, \mu_k)+\kappa\mu_k\} and (3.12) implies
\begin{equation} \begin{aligned} \sum\limits_{k\notin\mathcal{P}^s}\mu_k^2\leq\frac{1}{\alpha}(\widetilde{\mathcal{W}}(\mathbf{x}^0,\mu_{-1})+\kappa\mu_{-1}-\min\limits_\Omega\mathcal{W}). \end{aligned} \end{equation} | (3.25) |
Combining (3.24) and (3.25), the proof of (ii) is completed.
Theorem 3.5. If there is an accumulation point in \{\mathbf{x}^k:k\in\mathcal{P}^s\} , then the accumulation point is a lifted stationary point of (1.7).
Proof. Since (3.9) fails for k\in\mathcal{P}^s , by rearranging (3.21), we obtain that
\gamma_{k}\mu_{k}^{-1}\|\mathbf{x}^{k+1}-\mathbf{x}^{k}\|^2\leq2\alpha\mu_{k}^2, |
which gives
\|\mathbf{x}^{k+1}-\mathbf{x}^{k}\|\leq\sqrt{2\alpha\gamma_{k}^{-1}\mu_{k}^3}. |
Thus,
\gamma_{k}\mu_{k}^{-1}\|\mathbf{x}^{k+1}-\mathbf{x}^{k}\|\leq\sqrt{2\alpha\gamma_{k}\mu_{k}}, |
which together with \lim_{k\rightarrow \infty}\mu_{k} = 0 and \{\gamma_{k}\}\subseteq[\underline{\gamma}, \max\{\bar{\gamma}, \rho L\}] implies
\begin{equation} \begin{aligned} \lim\limits_{k\rightarrow \infty}\gamma_{k}\mu_{k}^{-1}\|\mathbf{x}^{k+1}-\mathbf{x}^{k}\| = 0 \; \; \mathrm{and} \; \lim\limits_{k\rightarrow \infty}\|\mathbf{x}^{k+1}-\mathbf{x}^k\| = 0. \end{aligned} \end{equation} | (3.26) |
Let \bar{\mathbf{x}} be an accumulation point of \{\mathbf{x}^k\}_{k\in\mathcal{P}^s} , (3.26) indicates that \{\mathbf{x}^k\} exists a subsequence \{\mathbf{x}^{k_t}\}_{k_t\in\mathcal{P}^s} converges to \bar{\mathbf{x}} . Similar analysis can be given for the case that k_t\in\mathcal{P}^s implies
\begin{equation} \begin{aligned} \lim\limits_{t\rightarrow \infty}\gamma_{k_t}\mu_{k_t}^{-1}\|\mathbf{x}^{k_t+1}-\mathbf{x}^{k_t}\| = 0 \; \; \mathrm{and} \; \lim\limits_{t\rightarrow \infty}\mathbf{x}^{k_t+1} = \bar{\mathbf{x}}. \end{aligned} \end{equation} | (3.27) |
Recalling \mathbf{x}^{k_t+1} = \hat{\mathbf{x}}^{k_t+1} defined in (3.6) and by its first-order necessary optimality condition, we have
\begin{equation} \begin{aligned} &\langle\nabla\tilde{f}(\mathbf{x}^{k_t},\mu_{k_t})+\gamma_{k_t}\mu_{k_t}^{-1}(\mathbf{x}^{k_t+1}-\mathbf{x}^{k_t})+\lambda\zeta^{k_t},\mathbf{x}-\mathbf{x}^{k_t+1}\rangle\geq0,\\ & \ \ \ \ \forall\zeta^{k_t}\in\partial\Phi^{d^{k_t}}(W\mathbf{x}^{k_t+1}),\mathbf{x}\in\Omega. \end{aligned} \end{equation} | (3.28) |
Since the elements in \{d^{k_t}:t\in\mathbb{N}\} are finite and \lim_{t\rightarrow \infty}\mathbf{x}^{k_t+1} = \bar{\mathbf{x}} , there exists a subsequence of \{k_t\} , denoted as \{k_{t_j}\} , and \bar{d}\in\mathbb{D}^p(W\bar{\mathbf{x}}) such that d^{k_{t_j}} = \bar{d}, \forall j\in\mathbb{N} . By the upper semicontinuity of \partial\Phi^{\bar{d}} and \lim_{j\rightarrow \infty}\mathbf{x}^{k_{t_j}+1} = \bar{\mathbf{x}} , it gives
\begin{equation} \begin{aligned} \left\{\lim\limits_{j\rightarrow \infty}\zeta^{k_{t_j}}:\zeta^{k_{t_j}}\in\partial\Phi^{d^{k_{t_j}}}(W\mathbf{x}^{k_{t_j}+1})\right\}\subseteq\partial\Phi^{\bar{d}}(W\bar{\mathbf{x}}). \end{aligned} \end{equation} | (3.29) |
Along with the subsequence \{k_{t_j}\} and letting j\rightarrow \infty in (3.28), from Definition 3.1-(ⅲ), (3.27) and (3.29), we obtain that there exist \bar{\xi}\in\partial f(\bar{\mathbf{x}}) and \bar{\zeta}^{\bar{d}}\in\partial\Phi^{\bar{d}}(W\bar{\mathbf{x}}) such that
\begin{equation} \begin{aligned} \langle\bar{\xi}+\lambda\bar{\zeta}^{\bar{d}},\mathbf{x}-\bar{\mathbf{x}}\rangle\geq0,\forall \mathbf{x}\in\Omega. \end{aligned} \end{equation} | (3.30) |
By \bar{d}\in\mathbb{D}^p(W\bar{\mathbf{x}}) , thanks to the convexity of f+\lambda\Phi^{\bar{d}} , (3.30) implies
f(\mathbf{x})+\lambda\Phi^{\bar{d}}(W\mathbf{x})-f(\bar{\mathbf{x}})-\lambda\Phi^{\bar{d}}(W\bar{\mathbf{x}}) \geq\langle\bar{\xi}+\lambda\bar{\zeta}^{\bar{d}},\mathbf{x}-\bar{\mathbf{x}}\rangle\geq0, \forall \mathbf{x}\in\Omega, |
which implies that \bar{\mathbf{x}} is a lifted stationary point of (1.7).
The purpose of this part is to test and verify the theoretical results and the properties of the SGD algorithm by the numerical experiments. We present Examples 1 and 2, which are respectively an under-determined linear regression problem and an over-determined censored regression problem. Especially, the process of solving subproblem (3.6) is very similar to the algorithm process of solving the LASSO problem.
All experiments are performed in MATLAB 2016a on a Lenovo PC with an Intel(R) Core(TM) i5-8250U CPU @1.60GHz 1801 Mhz and 8GB RAM. In the following examples, stopping criterion is set as
\begin{equation} \begin{aligned} \mathrm{number \; of\; iterations} \leq \mathbf{\mathtt{Maxiter}}\; \; \; \mathrm{or}\; \; \; \mu_k\leq\varepsilon. \end{aligned} \end{equation} | (4.1) |
We stop the proposed algorithm if the number of iterations exceeds \mathbf{\mathtt{Maxiter}} or the smoothing parameter is less than \varepsilon . Denote \bar{\mathbf{x}} the output of iterate \mathbf{x}^k . Set the fixed parameter \alpha = 1 throughout the numerical experiments.
Example 4.1. (Linear regression problem) Linear regression problems have been widely used in information theory[1], signal processing [35,36] and image restoration[6,36]. As pointed out in [20], \mathcal{\ell}_1 loss function is nonsmooth, but more robust and has stronger capability of outlier-resistance than the least squares loss function in the linear regression problems. Then we consider the following \mathcal{\ell}_0 regularized linear regression problem with \mathcal{\ell}_1 loss function:
\begin{equation} \begin{aligned} \min\limits_ {\mathbf{x}\in\Omega} \; \; \; \mathcal{W}_{\mathcal{\ell}_0}(\mathbf{x}): = \frac{1}{m}\|A\mathbf{x}-b\|_1+\lambda\|W\mathbf{\mathbf{x}}\|_0, \end{aligned} \end{equation} | (4.2) |
where A\in \mathbb{R}^{m\times n} with m = n , b\in\mathbb{R}^m . A smoothing function of the \mathcal{\ell}_1 loss function can be defined by
\begin{equation} \begin{aligned} &\tilde{f}(\mathbf{x},\mu) = \frac{1}{m}\sum\limits_{i = 1}^m\tilde{\theta}(A_i\mathbf{x}-b_i,\mu)\; \; with \; \; \tilde{\theta}(s,\mu) = \begin{cases} |s| & if \; |s| > \mu,\\ \frac{s^2}{2\mu}+\frac{\mu}{2} & if \; |s|\leq\mu.\\ \end{cases} \end{aligned} \end{equation} | (4.3) |
Denote s the \mathcal{\ell}_0 norm of true solution \mathbf{x}^* , i.e., \|W\mathbf{x}^*\|_0 = s . For the given positive integers m, n and s , the data are generated by
{\mathbf{\mathtt{W = randn(p,n)}}; \mathbf{\mathtt{B = randn(n,m)}}; \mathbf{ \mathtt{A = orth(B)'}}; \; \mathbf{\mathtt{b = A* \mathbf{x}^* +0.01*randn(m,1)}}.} |
In the algorithm, we set the parameters as below: \underline{\gamma} = \overline{\gamma} = 1, \; \mu_0 = 3.533, \; \mathbf{\mathtt{Maxiter}} = 10^3, \; \nu = 35.6014, \; \sigma = 3.0003, \; \rho = 1.0001, \; \kappa = \frac{1}{2}. Generate A, b and \mathbf{x}^* with m = n = 45 , p = 45 and s = 2 , set \lambda = 10^{-3} in (4.2) and \varepsilon = 10^{-3} in the stopping criterion (4.1). We set \mathbf{x}_0 = ones(n, 1) . Figure 1 shows the numerical results. Figure 1 plots \mathbf{x}^* and \overline{\mathbf{x}} , where \mathbf{x}^* and \overline{\mathbf{x}} denote the original signal (which can also be expressed as true solution) and the output of iterate \mathbf{x}^k from the SGD algorithm. From Figure 1, we can see that the output of \mathbf{x}^k is very close to the original generated signal.
Now we use another form of matrix W to solve Example 4.1:
W = \begin{pmatrix} 0& 0 & 0 & \cdots &0& 0 \\ -3 & 1 & 0 & \cdots &0& 0 \\ 0 &-3 & 1 & \cdots &0& 0 \\ \vdots & \vdots & \vdots & \ddots & \vdots& \vdots\\ 0 & 0 & 0 & \cdots &-3& 1 \end{pmatrix}_{p \times n}. |
Set \underline{\gamma} = \overline{\gamma} = 1, \; \mu_0 = 3.533, \; \mathbf{\mathtt{Maxiter}} = 10^3, \; \nu = 36, \; \sigma = 7, \; \rho = 1.0001 \; \mathrm{and}\; \kappa = \frac{1}{2}. We randomly generate the data as follows:
{\mathbf{\mathtt{B = randn(n,m)}};\mathbf{ \mathtt{A = orth(B)'}}; \; \mathbf{\mathtt{b = A* \mathbf{x}^* +0.01*randn(m,1)}}.} |
We run numerical experiments with (m, n, p, s) = (45, 45, 45, 2). Set \lambda = 10^{-3} in (4.2) and \varepsilon = 10^{-3} in the stopping criterion (4.1). We define \mathbf{x}_0 = randn(n, 1) . From Figure 2, we can see that the output of \mathbf{x}^k obtained by the SGD algorithm is also close to the true solution \mathbf{x}^* .
The last special case of W is the penalty matrix in 1-dimensional Fused LASSO [39]:
W = \begin{pmatrix} -1& 1 & 0 & \cdots &0& 0 \\ 0 & -1 & 1 & \cdots &0& 0 \\ 0 &0 & -1 & \cdots &0& 0 \\ \vdots & \vdots & \vdots & \ddots & \vdots& \vdots\\ 0 & 0 & 0 & \cdots &-1& 1 \end{pmatrix}_{p \times n}. |
Set \nu = 40, \; p = 45, \; n = 46 and \mathbf{x}_0 = ones(n, 1) . The remaining parameter values are the same as the previous situation (see Figure 3). From Figures 1–3, we can see that the output of \mathbf{x}^k obtained by the SGD algorithm is close to the true solution \mathbf{x}^* .
Example 4.2 (Censored regression problem) The application of censored regression problems has been studied in machine learning[37], economics[38], biomedical, and technical systems. The following censored regression problem is also a typical class of composite sparse optimization problems with nonsmooth convex loss functions. Now we consider the following {\mathcal{\ell}_0} regularized censored regression problem:
\min \limits_ {\mathbf{x}\in\Omega}\; \; \; \mathcal{W}_{\mathcal{\ell}_0}(\mathbf{x}): = \frac{1}{m}\|\max \{A\mathbf{x},0\}-b\|_1+\lambda\|W\mathbf{\mathbf{x}}\|_0, |
where A\in\mathbb{R}^{m\times n} and b\in\mathbb{R}^m . For the loss function in (1.5), a smoothing function of it can be defined by
\begin{equation*} \tilde{f}(\mathbf{x},\mu) = \frac{1}{m}\sum\limits_{i = 1}^m\tilde{\theta}(\tilde{\phi}(A_i\mathbf{x},\mu)-b_i,\mu)\; \; with \; \; \tilde{\phi}(s,\mu) = \begin{cases} \max\{s,0\} & if \; |s| > \mu,\\ \frac{(s+\mu)^2}{4\mu} & if \; |s|\leq\mu.\\ \end{cases} \end{equation*} |
Set \varepsilon = 10^{-2} , \nu = 16.0009 , \lambda = 10^{-3} , \mu_0 = 10.8999 , \sigma = 4.0003 , \kappa = \frac{1}{2} , \rho = 1.2006 and \mathbf{x}_0 = randn(n, 1) . In this example, we run numerical experiments with (m, n, p, s) = (40, 40, 40, 2), we randomly generate the problem data as follows:
\mathbf{\mathtt{A = randn(m,n)}}; \; \mathbf{\mathtt{W = randn(p,n)}}; \; \mathbf{ \mathtt{b = max(A* \mathbf{x}^* +0.01*randn(m,1),0)}}. |
The computational results of \mathbf{x}^* and \mathbf{x} are shown in Figure 4.
We use the following form of W to solve Example 4.2:
W = \begin{pmatrix} 0& -2 & 0 & 0&\cdots &0& 0 \\ -2 & 1 & -3 & 0&\cdots &0& 0 \\ 0 &-3 & 1 & -4&\cdots &0& 0 \\ \vdots & \vdots & \vdots & \ddots & \vdots& \vdots& \vdots \\ 0 & 0 & 0 & 0&\cdots &1& -m\\ 0 & 0 & 0 & 0&\cdots &-m& 1 \end{pmatrix}_{p\times n}. |
Set \underline{\gamma} = \overline{\gamma} = 3, \; \mu_0 = 2, \; \mathbf{\mathtt{Maxiter}} = 10^3, \; \nu = 1.2200, \; \sigma = 2.6915, \; \rho = 1.0001 \; \mathrm{and}\; \kappa = 0.711. For the given positive integers m, n and s , the data are generated by
\mathbf{\mathtt{A = randn(m,n)}}; \; \mathbf{\mathtt{b = A* \mathbf{x}^* +0.01*randn(m,1)}}. |
We run numerical experiments with (m, n, p, s) = (40, 40, 40, 2). Set \lambda = 10^{-3} in (4.2), \mathbf{x}_0 = ones(n, 1) and \varepsilon = 10^{-3} . From Figures 4 and 5, it can be seen that the output of \mathbf{x}^k is very close to the true solution.
We have intensively studied the composite sparse optimization problem consisting of the sum of a nonsmooth convex function and the {\mathcal{\ell}_0} penalty term of a matrix times the coefficient vector. Considering the original cardinality penalty problem and an exact continuous relaxation problem with capped- {\mathcal{\ell}_1} penalty, we have proved several novel and interesting results: the consistency between global minimizers of the relaxation problem and the original problem, and local minimizers of relaxation problems are local minimizers of the original problem. We propose the SGD algorithm based on the smoothing method and the smoothing gradient descent algorithm. Then SGD algorithm has been investigated from both a theoretical and an algorithmic point of view. So we prove that if the sequence generated by the algorithm has an accumulation point, then it is a lifted stationary point of relaxation problem. This well explains why the algorithm is expected to enjoy an appealing performance from the theoretical perspective, which is testified by the numerical experiments. Our initial numerical results confirm the predicted underlying theoretical results.
Wei Yang: Conceptualization, writing-original draft, validation, software; Lili Pan: Conceptualization, supervision, funding acquisition, validation, software; Jinhui Wan: Software, methodology, data curation. All authors have read and approved the final version of the manuscript for publication.
The work of the authors was supported by the National Natural Science Foundation of China grants 12271309.
The authors declare no conflicts of interest.
[1] |
P. Deligne, La conjecture de Weil. Ⅰ, Publ. Math. Inst. Hautes Études Sci., 43 (1974), 273–307. https://doi.org/10.1007/BF02684373 doi: 10.1007/BF02684373
![]() |
[2] |
E. Hecke, Theorie der eisensteinsche reihen höherer stufe und ihre anwendung auf funktionen-theorie und arithmetik, Abhandlungen Math. Semin. Univ. Hamb., 5 (1927), 199–224. https://doi.org/10.1007/BF02952521 doi: 10.1007/BF02952521
![]() |
[3] |
J. Wu, Power sums of Hecke eigenvalues and application, Acta Arith., 137 (2009), 333–344. https://doi.org/10.4064/aa137-4-3 doi: 10.4064/aa137-4-3
![]() |
[4] |
R. A. Rankin, Contributions to the theory of Ramanujan's function \tau(n) and similar arithmetical functions, Math. Proc. Cambridge Philos. Soc., 35 (1939), 351–372. https://doi.org/10.1017/S0305004100021101 doi: 10.1017/S0305004100021101
![]() |
[5] | A. Selberg, V. Bjerknes, J. Molland, Bemerkungen über eine Dirichletsche Reihe, die mit der theorie der modulformen nahe verbunden ist, Cammermeyer i Komm, 1940. |
[6] |
B. R. Huang, On Rankin-Selberg problem, Math. Ann., 381 (2021), 1217–1251. https://doi.org/10.1007/s00208-021-02186-7 doi: 10.1007/s00208-021-02186-7
![]() |
[7] |
Y. K. Lau, G. S. Lü, Sums of Fourier coefficients of cusp forms, Q. J. Math., 62 (2011), 687–716. https://doi.org/10.1093/qmath/haq012 doi: 10.1093/qmath/haq012
![]() |
[8] |
C. R. Xu, General asymptotic formula of Fourier coefficients of cusp forms over sum of two squares, J. Number Theory, 236 (2022), 214–229. https://doi.org/10.1016/j.jnt.2021.07.017 doi: 10.1016/j.jnt.2021.07.017
![]() |
[9] |
S. Zhai, Average behavior of Fourier coefficients of cusp forms over sum of two squares, J. Number Theory, 133 (2013), 3862–3876. https://doi.org/10.1016/j.jnt.2013.05.013 doi: 10.1016/j.jnt.2013.05.013
![]() |
[10] |
H. F. Liu, On the asymptotic distribution of Fourier coefficients of cusp forms, Bull. Braz. Math. Soc., 54 (2023), 21. https://doi.org/10.1007/s00574-023-00335-x doi: 10.1007/s00574-023-00335-x
![]() |
[11] |
S. Manski, J. Mayle, N. Zbacnik, The asymptotic distribution of a hybrid arithmetic function, Integers, 15 (2015), A28. https://doi.org/10.5281/zenodo.10456207 doi: 10.5281/zenodo.10456207
![]() |
[12] | L. L. Wei, H. X. Lao, The mean value of a hybrid arithmetic function associated to Fourier coefficients of cusp forms, Integers, 19 (2019), A44. |
[13] | H. Iwaniec, Topics in classical automorphic forms, Providence, Rhode lsland: American Mathematical Society, 1997. https://doi.org/10.1090/gsm/017 |
[14] |
J. Newton, J. A. Thorne, Symmetric power functoriality for holomorphic modular forms, Publ. Math. l'IHÉS, 134 (2021), 1–116. https://doi.org/10.1007/s10240-021-00127-3 doi: 10.1007/s10240-021-00127-3
![]() |
[15] |
J. Newton, J. A. Thorne, Symmetric power functoriality for holomorphic modular forms, Publ. Math. l'IHÉS, 134 (2021), 117–152. https://doi.org/10.1007/s10240-021-00126-4 doi: 10.1007/s10240-021-00126-4
![]() |
[16] |
A. Perelli, General L-functions, Ann. Mat. Pura Appl., 4 (1982), 287–306. https://doi.org/10.1007/BF01761499 doi: 10.1007/BF01761499
![]() |
[17] |
J. Bourgain, Decoupling, exponential sums and the Riemann zeta function, J. Amer. Math. Soc., 30 (2017), 205–224. https://doi.org/10.1090/jams/860 doi: 10.1090/jams/860
![]() |
[18] |
Y. Lin, R. Nunes, Z. Qi, Strong subconvexity for self-dual GL(3) L-functions, Int. Math. Res. Not., 2023 (2023), 11453–11470. https://doi.org/10.1093/imrn/rnac153 doi: 10.1093/imrn/rnac153
![]() |
[19] | K. Matsumoto, The mean values and the university of Rankin-Selberg L-functions, Number Theory, 2001,201–221. |
[20] | K. Ramachandra, A. Sankaranarayanan, Notes on the Riemann zeta-function, J. Indian Math. Soc., 57 (1991), 67–77. |
[21] | A. Ivić, Exponent pairs and the zeta function of Riemann, Studia Sci. Math. Hunger., 15 (1980), 157–181. |
[22] | G. Tenenbaum, Introduction to analytic and probabilistic number theory, Cambridge University Press, 1995. https://doi.org/10.1090/gsm/163 |