Fractional $p$-sub-Laplacian operator problem with concave-convex nonlinearities on homogeneous groups

1.   Introduction

We consider the following $p$-fractional Laplace equation

where $\Omega$ is an open bounded domain in homogeneous Lie group $\mathbb{G}$ with smooth boundary, $p>1$, the parameter $\lambda>0$, $f$ and $h$ are sign-changing smooth functions, $1< \alpha< p<\beta<p^*_{s}: = \frac{Qp}{Q-ps}$, $p^*_{s}$ is the fractional critical Sobolev exponent in this context and $Q>sp$ is the homogeneous dimension of the homogeneous Lie group $\mathbb{G}$. The operator $(-\Delta_{p,g})^{s}$ is the fractional $p$-sub-Laplacian operator on $\mathbb{G}$ which is defined by

where $\mathcal{B}_{g}(x,\varepsilon)$ is the quasi-ball of center $x\in\mathbb{G}$ and radius $\varepsilon>0$ with respect to the homogeneous quasi-norm $g$. In our work, the homogeneous quasi-norm $g:\mathbb{G}\to\mathbb{R}^+_0$ is a continuous function satisfying the following properties:

(ⅰ) $g(x) = 0\ \rm{if\;and\;only\;if\;} x = 0$ for every $x\in\mathbb{G}$;

(ⅱ) $g(x^{-1}) = g(x)$ for every $x\in\mathbb{G}$;

(ⅲ) $g(\delta_{\mu}(x)) = \mu g(x)$ for every $\mu>0$ and for every $x\in\mathbb{G}$, where $\delta_{\mu}$ is a dilations on homogeneous Lie group $\mathbb{G}$.

Associated with (1.1), we have the energy functional $I_{\lambda}: \, E^{0}_{g}\to \mathbb{R}$ defined by

By a direct calculation, we have that $I_{\lambda}\in C^{1}(E^{0}_{g},\mathbb{R})$ and

where $E_{g}^{0}$ is a subspace of $E_{g}$ defined as $E_{g}^{0} = \{u\in E_{g}:\,\,u = 0 \,\,\,\rm{in}\,\,\,\mathbb{G}\backslash\Omega\}$ with the norm

Here $\mathcal{Q} = \mathbb{G}^{2} \backslash(\mathcal{C}_{\Omega}\times \mathcal{C}_{\Omega})$ and $\mathcal{C}_{\Omega} = \mathbb{G}\backslash\Omega$. See Section 2 for more details.

Recently, a lot of attention is given to the study of fractional operators of elliptic type due to concrete real world applications in finance, thin obstacle problem, optimization, quasi-geostrophic flow etc. Dirichlet boundary value problem in case of fractional Laplacian with polynomial type nonlinearity using variational methods is recently studied in [4,6,21,20,19]. For example, Brändle et. al [4] studied the fractional Laplacian operator $(-\Delta)^{s}$ equation involving concave-convex nonlinearity for the subcritical case in the Euclidean space $\mathbb{R}^{N}$, they prove that there exists a finite parameter $\Lambda>0$ such that for each $\lambda\in (0,\Lambda)$ there exist at least two solutions, for $\lambda = \Lambda$ there exists at least one solution and for $\lambda\in (\Lambda,+\infty)$ there is no solution. Barrios et al. [2] studied the non-homogeneous equation involving fractional Laplacian and proved the existence and multiplicity of solutions under suitable conditions of $s$ and $q$. Zhang, Liu and Jiao [23] studied the fractional equation with critical Sobolev exponent, they proved that the existence and multiplicity of solutions under appropriate conditions on the size of $\lambda$. For more other advances on this topic, see [19] for the subcritical, [20] for the critical case, [22] for the supercritical case, and fractional Laplacian equation with Hardy-type potential are shown in [13,14,24,25,26,27]. Moreover, for the fractional $p$-Laplacian equation, eigenvalue problem related to $p$-fractional Laplacian is studied in [17,12. Goyal and Sreenadh [15] studied the fractional $p$-Laplacian equation involving concave-convex nonlinearities. By using the Nehari manifold and the fibering maps methods, they showed that the problem has at least two non-negative solutions.

In this paper, we present results concerning fractional forms Laplacian operator on homogeneous Lie groups. As usual, the general approach based on homogeneous Lie groups allows one to get insights also in the Abelian case, for example, from the point of view of the possibility of choosing an arbitrary quasi-norm. Moreover, another application of the setting of homogeneous Lie groups is that the results can be equally applied to elliptic and subelliptic problems. We start by discussing fractional Sobolev inequalities on the homogeneous Lie groups. As a consequence of these inequalities, we derive the existence results for the nonlinear problem with fractional $p$-sub-Laplacian operator and concave-convex nonlinearities and sign-changing weight functions. We also extend this analysis to equations of fractional $p$-sub-Laplacians and to Riesz type potential operators.

To the best of our knowledge there is no work for fractional $p$-sub-Laplacian operator with convex-concave type nonlinearity and sign changing weight functions on the homogeneous Lie groups. We have the following existence result.

Theorem 1.1. Let $\mathbb{G}$ be a homogeneous Lie group with homogeneous dimension $Q$, and let $s\in (0, 1)$, $Q > sp$, $1<\alpha<p<\beta<p^*_{s}$ and $f\in L^{\frac{p^*_{s}}{p^*_{s}-\alpha}}(\Omega)$, $h\in L^{\frac{p^*_{s}}{p^*_{s}-\beta}}(\Omega)$, $f^{\pm} = \max\{\pm f,0\}\neq 0$, $h^{\pm} = \max\{\pm h,0\}\neq 0$. Then there exists $\Lambda_{*}>0$ such that the equation (1.1) admits at least two non-negative solutions for $\lambda\in(0,\Lambda_*)$.

The paper is organized as follows: In Section 2, we study the properties of the Sobolev spaces $W^{s,p}_{g}(\mathbb{G})$ and $E^{0}_{g}$ on homogeneous groups. In Section 3, we introduce Nehari manifold and study the behavior of Nehari manifold by carefully analyzing the associated fibering maps on homogeneous Lie groups. Section 4 contains the existence of nontrivial solutions in $\mathcal{N}_{\lambda}^{+}$ and $\mathcal{N}_{\lambda}^{-}$.

2.   Functional analytic settings on homogeneous Lie groups

In this section we discuss nilpotent Lie algebras and groups in the spirt of Folland and Stein's book [11] as well as introducing homogeneous Lie groups. For more analyses and details in this direction we refer to the recent open access book [10] and [1,5,3,7,8,9,18] and references therein.

Let $\mathfrak{g}$ be a real and finite-dimensional Lie algebra, and let $\mathbb{G}$ be the corresponding connected and simply-connected Lie group. The lower central series of $\mathfrak{g}$ is defined inductively by

If $\mathfrak{g}_{(s+1)} = \{0\}$ and $\mathfrak{g}_{(s)}\neq \{0\}$, then $\mathfrak{g}$ is said to be nilpotent of step $s$. A Lie group $\mathbb{G}$ is nilpotent of step $s$ whenever its Lie algebra is nilpotent of step $s$.

Let $\exp:\mathfrak{g}\to \mathbb{G}$ be a exponential map, and $\mathbb{G}$ be a connected and simply-connected nilpotent Lie group with Lie algebra $\mathfrak{g}$. Then, exponential map $\exp$ is a diffeomorphism from $\mathfrak{g}$ to $\mathbb{G}$. Let $A$ be a diagonalisable linear operator on $\mathfrak{g}$ with positive eigenvalues, and $\mu>0$. Define the mappings are of the form

Then, let us give a family of dilations of a Lie algebra $\mathfrak{g}$ as follow

which satisfies:

(ⅰ) $\delta_{\mu}$ is a morphism of the Lie algebra $\mathfrak{g}$, that is, a linear mapping from $\mathfrak{g}$ to itself which respects to the Lie bracket:

(ⅱ) $\delta_{\mu_{1}\mu_{2}} = \delta_{\mu_{1}}\delta_{\mu_{2}}$ for all $\mu_{i}>0$, $i = 1,2$. If $k>0$ and $\{\delta_{\mu}\}$ is a family of dilations on $\mathfrak{g}$, then so is $\{\widetilde{\delta}_{\mu}\}$, where $\widetilde{\delta}_{\mu} = \delta_{\mu^{k}} = \rm{exp}(k A\log \mu).$

Remark 2.1. ${(i)}$ If a Lie algebra $\mathfrak{g}$ admits a family of dilations, then it is nilpotent, but not all nilpotent Lie algebras admit a dilation structure.

${(ii)}$ Since the exponential mapping $\exp$ is a global diffeomorphism from $\mathfrak{g}$ to $\mathbb{G}$, it induces the corresponding family on $\mathbb{G}$ which we may still call the dilations on $\mathbb{G}$ and denote by $\delta_{\mu}$. Thus, for $x\in\mathbb{G}$ we will write $\delta_{\mu}(x)$ or abbreviate it writing simply $\mu x$, and the origin of $\mathbb{G}$ will be usually denoted by $0$.

Definition 2.1. Let $\delta_{\mu}$ be a dilations on $\mathbb{G}$. We say that a Lie group $\mathbb{G}$ is a homogeneous Lie group if:

(a) It is a connected and simply-connected nilpotent Lie group $\mathbb{G}$ whose Lie algebra g is endowed with a family of dilations $\{\delta_{\mu}\}$.

(b) The maps $\exp\circ\delta_{\mu}\circ \exp^{-1}$ are group automorphism of $\mathbb{G}$.

Now, we give some two examples of homogeneous groups.

Example 2.1. The Euclidean space $\mathbb{R}^N$ is a homogeneous group with dilation given by the scalar multiplication.

Example 2.2. If $N$ is a positive integer, the Heisenberg group $\mathbb{H}^N$ is the group whose underlying manifold is $\mathbb{C}^N\times\mathbb{R}$ and whose multiplication is given by

where $(z,t) = (z_{1},\cdots,z_{N},t) = (x_{1},y_{1},\cdots,x_{N},y_{N},t)\in \mathbb{H}^{N}$, $x\in \mathbb{R}^{N}$ $y\in \mathbb{R}^{N}$ and $t\in \mathbb{R}$. The Heisenberg group $\mathbb{H}^N$ is a homogeneous group with dilations

The mappings $\{\delta_{\mu}\}$ give the dilation structure to an $N$-dimensional homogeneous Lie group $\mathbb{G}$ with

where $(x_{1},\cdots,x_{N})$ are the exponential coordinates of $x\in \mathbb{G}$, $d_{j}\in \mathbb{N}$ for every $j = 1,2,\cdots,N$ and $1 = d_{1} = \cdots = d_{m}< d_{m+1}\leq \cdots\leq d_{N}$ for $m: = \rm{dim}(V_1)$. Here the group $\mathbb{G}$ and the algebra $\mathfrak{g}$ are identified through the exponential mapping.

It is customary to denote with $Q: = \sum_{i = 1}^{k}i\cdot\rm{dim}(V_i)$ the homogeneous dimension of $\mathbb{G}$ which corresponds to the Hausorff dimension of $\mathbb{G}$. From now on $Q$ will always denote the homogneous dimension of $\mathbb{G}$. For example, the homogeneous dimension of Heisenberg group $\mathbb{H}^{N}$ is $Q: = 2N+2$.

Now, we define a homogeneous quasi-norm on a homogeneous Lie group $\mathbb{G}$ to be a continuous function $g:\mathbb{G}\to \mathbb{R}^{+}$ with the following properties:

(ⅰ) $g(x) = 0\ \rm{if and only if } x = 0$ for every $x\in\mathbb{G}$;

(ⅱ) $g(x^{-1}) = g(x)$ for every $x\in\mathbb{G}$;

(ⅲ) $g(\delta_{\mu}(x)) = \mu g(x)$ for every $\mu\in\mathbb{R}^+$ and for every $x\in\mathbb{G}$.

For any measurable set $E\subset \mathbb{G}$, we have $|\delta_{\mu}(E)| = \mu^{Q}|E|$, $d(\delta_{\mu}(x)) = \mu^{Q}dx$, where $|E|$ denotes the measure of the set $E$. Let

be the quasi-ball of radius $r>0$ about $x$ with respect to the homogeneous quasi-norm $g$. Then, we have that

It can be noticed that $\mathcal{B}_{g}(x,r)$ is the left translate by $x$ of $\mathcal{B}_{g}(0,r)$, which in turn is the image under $\delta_{r}$ of $\mathcal{B}_{g}(0,1)$. Moreover, let

be the unit sphere with respect to the homogeneous quasi-norm $g$. Then there is a unique positive Radon measure $\sigma$ on $\mathcal{S}_{g}(0)$ such that for all $f\in L^{1}(\mathbb{G})$, we have

Let $\mathbb{G}$ be a homogeneous Lie group, with its basis $X_1$, ..., $X_{N}$, generating its Lie algebra $\mathfrak{g}$ through their commutators. Then, the sub-Laplacian operator is defined as

In the sequel, we use the following notations for the horizontal gradient

and for the horizontal divergence

Using the Green's first and second formulae, we can define the $p$-sub-Laplacian on homogeneous groups $\mathbb{G}$ as

Recently, a great deal of attention has been focused on studying of equations or systems involving fractional Laplacian and corresponding nonlocal problems, both for their interesting theoretical structure and for their concrete applications, see [6,17,19,20] and references therein. The fractional $p$-Laplacian operator $(-\Delta_{p})^{s}$, $s\in (0,1)$, is defined as

This type of operator arises in a quite natural way in many different contexts, such as, the thin obstacle problem, finance, phase transitions, anomalous diffusion, flame propagation and many others.

Let $\mathbb{G}$ be a homogeneous Lie group with homogeneous dimension $Q$, $p>1$ and $s\in(0,1)$. Compared to the fractional $p$-Laplacian problem, the fractional $p$-sub-Laplacian $(-\Delta_{p,g})^{s}$ on $\mathbb{G}$ can be defined as

where $g$ is a quasi-norm on $\mathbb{G}$ and $\mathcal{B}_{g}(x,\varepsilon)$ is a quasi-ball with respect to $g$, with radius $\varepsilon$ centered at $x\in\mathbb{G}$.

Now we recall the definitions of the fractional Sobolev spaces on homogeneous Lie groups $\mathbb{G}$. For a measurable function $u:\mathbb{G}\rightarrow \mathbb{R}$ we define the Gagliardo quasi-seminorm by

For $p>1$ and $s\in (0,1)$, we introduce the the functional Sobolev space on homogeneous Lie groups $\mathbb{G}$ by

and endowed with the norm

Similarly, if $\Omega\subset \mathbb{G}$ is a Haar measurable set, we define the Sobolev space

endowed with norm

In [16,Theorem 2], Kassymov and Suragan given the following analogue of the fractional Sobolev inequality on homogeneous groups $\mathbb{G}$.

Theorem 2.1. Let $\mathbb{G}$ be a homogeneous group with homogeneous dimension $Q$. Assume that $p > 1$, $s\in(0, 1)$, $Q > sp$ and $g$ denotes a quasi-norm on $\mathbb{G}$. Then, for any measurable and compactly supported function $u : \mathbb{G}\to \mathbb{R}$, there exists a positive constant $C = C(Q,p,s)> 0$ such that

where $p^{*}_{s}: = p^*(Q,s,p) = \frac{Qp}{Q-sp}$ is the fractional critical Sobolev exponents on homogeneous group.

Let $\mathcal{Q} = \mathbb{G}^{2} \backslash(\mathcal{C}_{\Omega}\times \mathcal{C}_{\Omega})$ and $\mathcal{C}_{\Omega} = \mathbb{G}\backslash\Omega$. We define the Sobolev space

endowed with the norm as following

Indeed, if $\|u\|_{E_g} = 0$, we get that $\|u\|_{L^{p}(\Omega)} = 0$ and $\iint_{\mathcal{Q}}\frac{|u(x)- u(y)|^{p}}{g(y^{-1}\circ x)^{Q+sp}}dxdy = 0$. Then, the above equalities imply that $u = 0$ a.e. in $\Omega$ and $u(x) = u(y) = const.$ a.e. in $\mathcal{Q}$. So, we can get that $u = 0$ a.e. in $\mathbb{G}$ and $\|\cdot\|_{E_g}$ is a norm on $E_{g}$.

Let $E_{g}^{0} = \{u\in E_{g}:u = 0\,\,\,\rm{in}\,\,\,\mathbb{G}\backslash \Omega\}$ be a subspace of $E_{g}$. Then, for any $p>1$, $E^{0}_{g}$ is a Banach space and have the following properties.

Lemma 1. The following hold.

$(i)$ If $u\in E_{g}$, then $u\in W^{s, p}_{g}(\Omega)$ and $\|u\|_{W^{s,p}_{g}(\Omega)}\leq \|u\|_{E_{g}}$.

$(ii)$ If $u\in E_{g}^{0}$, then $u\in W^{s, p}_{g}(\mathbb{G})$ and $\|u\|_{W^{s,p}_{g}(\Omega)}\leq \|u\|_{W^{s,p}_{g}(\mathbb{G})} = \|u\|_{E_{g}}.$

Proof. (i) Let $u\in E_{g}$, since $\Omega\times\Omega$ is strictly contained in $\mathcal{Q}$, we have

Thus $\|u\|_{W^{s,p}_{g}(\Omega)}\leq \|u\|_{E_g}$ and deduced the result (ⅰ).

(ⅱ) For each $u\in E^{0}_{g}$, we get $u = 0$ on $\mathbb{G}\setminus \Omega$. Hence, $\|u\|_{L^{2}(\mathbb{G})} = \|u\|_{L^{2}(\Omega)}$ and

which and (2.3) yield the result (ⅱ).

Theorem 2.2. Let $\mathbb{G}$ be a homogeneous group with homogeneous dimension $Q$. Assume that $p > 1$, $s\in (0, 1)$, $Q > sp$ and $g$ be a quasi-norm on $\mathbb{G}$. Then for every $u\in E^{0}_{g}$ there exists a positive constant $c = c(Q,s)>0$ depending on $Q$ and $s$ such that

Proof. For any $u\in E^{0}_{g}$, by Lemma 2.1 (ⅱ) and Theorem 2.1, we know that $u\in W^{s,p}_{g}(\mathbb{G})$ and $W^{s,p}_{g}(\mathbb{G})\hookrightarrow L^{p^*_{s}}(\mathbb{G})$. Then, we have

and completes the proof of Theorem 2.2.

Lemma 2.2. The space $(E^{0}_{g}, \|\cdot\|_{E^{0}_{g}})$ is a reflexive Banach space.

Proof. Let $\{u_k\}_{k}\subset E^{0}_{g}$ be a Cauchy sequence. By Lemma 2.1 and Theorem 2.2, $\{u_k\}_{k}$ is Cauchy sequence in $L^{p}(\Omega)$ and so $\{u_k\}_{k}$ has a convergent subsequence. We assume $u_k\to u$ in $L^{p}(\Omega)$. Since $u_k = 0$ in $\mathbb{G}\setminus \Omega$, we define $u = 0$ in $\mathbb{G}\setminus \Omega$ and then $u_k\to u$ strongly in $L^{p}(\mathbb{G})$ as $k\to \infty$. So, there exists a subsequence of $\{u_{k}\}_{k}$, still denoted by $\{u_{k}\}_{k}$, such that $u_{k}\to u$ a.e. in $\mathbb{G}$. By Fatou's Lemma and using the fact that $\{u_k\}$ is a Cauchy sequence, we get that $u\in E^{0}_{g}$ and $\|u_k -u\|_{E_{g}}\to 0$ as $k\to+\infty$. Hence $E^{0}_{g}$ is a Banach space. Reflexivity of $E^{0}_{g}$ follows from the fact that $E^{0}_{g}$ is a closed subspace of reflexive Banach space $W^{s,p}_{g}(\mathbb{G})$.

From above results, we can defined the following scalar product

and norm

for the reflexive Banach space $E_{g}^{0}$. Since $u = 0$ a.e. in $\mathbb{G}\backslash\Omega$, we note that the (2.5) and (2.6) can be extended to all $\mathbb{G}$. Moreover, for any $u\in E_{g}$, by Theorem 2.2 and the embedding $L^{p^*_{s}}(\Omega)\hookrightarrow L^{p}(\Omega)$, there exist $C_{1}$ and $C_{2}>0$ such that

This imply that the norm $\|\cdot\|_{E^{0}_{g}}$ on $E^{0}_{g}$ is equivalent to the norm $\|\cdot\|_{E_{g}}$ on $E_{g}$, and the norm $\|\cdot\|_{E^{0}_{g}}$ involves the interaction between $\Omega$ and $\mathbb{G}\setminus\Omega$. But the norms in (2.1) and (2.2) are not same because $\Omega\times\Omega$ is strictly contained in $\mathcal{Q}$.

Lemma 2.3. Let $\{u_k\}_{k}$ be a bounded sequence in $E^{0}_{g}$. Then, there exists $u\in L^{q}(\mathbb{G})$ such that $u_k\to u$ in $L^{q}(\mathbb{G})$ as $k\to \infty$ for any $q\in [1, p^{*}_{s})$.

Proof. Let $\{u_k\}_{k}$ is bounded in $E^{0}_{g}$, by Lemmas 2.1 and (2.7), $\{u_k\}_{k}$ is bounded in $W^{s,p}_{g}(\Omega)$ and also in $L^{p}(\Omega)$. Then by assumption on $\Omega$ and [4, Corollary 7.2], there exists $u\in L^{q}(\Omega)$ such that up to a subsequence $u_k\to u$ in $L^{q}(\Omega)$ as $k\to\infty$ for any $q\in[1,p^*_{s})$. Since $u_k = 0$ on $\mathbb{G}\setminus \Omega$, we can define $u: = 0$ in $\mathbb{G}\setminus \Omega$ and we get $u_k \to u$ in $L^{q}(\mathbb{G})$.

From Theorem 2.2, Lemma 2.1 and Lemma 2.3, we have that the embedding $E_{g}^{0}\hookrightarrow L^{q}(\Omega)$ is continuous for any $q\in [1,p^{*}_{s}]$ and compact whenever $q\in [1,p^{*}_{s})$. Let $S_{p^{*}_{s}}$ be the best constant for the embedding of $E_{g}^{0}\hookrightarrow L^{p^{*}_{s}}(\Omega)$ defined by

3.   The fibering properties

Since the energy functional $I_{\lambda}$ is not bounded below on the space $E^{0}_{g}$, we consider the Nehari minimization problem: for $\lambda>0$,

where

For any $u\in \mathcal{N}_{\lambda}$, the following equality hold

which implies that $\mathcal{N}_{\lambda}$ contains all nonzero solutions of equation (1.1). Moreover, we have the following result.

Lemma 3.1. $I_{\lambda}$ is coercive and bounded below on $\mathcal{N}_{\lambda}$ for all $\lambda>0$.

Proof. Let $\lambda>0$ and for all $u\in\mathcal{N}_{\lambda}$, from (3.1) and Theorem 2.2 there holds

which yields that $I_{\lambda}$ is bounded below and coercive on $\mathcal{N}_{\lambda}$ since $\beta>p>\alpha$ and $S_{\alpha^{*}_{s}}>0$. This completes the proof of Lemma 3.1.

Define

Then, we see that $\Phi_{\lambda}\in C^{1}(E_{g}^{0},\mathbb{R})$, $\mathcal{N}_{\lambda} = \Phi_{\lambda}^{-1}(0)\backslash\{0\}$, and for all $u\in \mathcal{N}_{\lambda}$ we get that

We split $\mathcal{N}_{\lambda}$ into three parts

On $\mathcal{N}_{\lambda}^{0}$, the following result hold.

Lemma 3.2. For each $\lambda>0$, let $u_{0}$ be a local minimizer for $I_{\lambda}$ on $\mathcal{N}_{\lambda}\backslash\mathcal{N}_{\lambda}^{0}$, then $u_{0}$ is a critical point of $I_{\lambda}$.

Proof. Since $u_{0}$ is a local minimizer for $I_{\lambda}$ on $\mathcal{N}_{\lambda}$, that is, $u_{0}$ is a solution of the optimization problem

Then, by the theory of Lagrange multipliers, there exists a constant $\theta\in\mathbb{R}$ such that

Since $u_{0}\not\in \mathcal{N}^{0}_{\lambda}$, we have $\langle \Phi'_{\lambda}(u_{0}),u_{0}\rangle \neq0$, thus $\theta = 0$, this completes the proof.

Remark 3.1. Lemmas 3.1 and 3.2 imply that the functional $I_{\lambda}$ is bounded below on an appropriate subset of $E^{0}_{g}$ and the minimizers of functional $I_{\lambda}$ on subsets $\mathcal{N}_{\lambda}^{+}$, $\mathcal{N}_{\lambda}^{-}$ giving raise to solutions of (1.1).

Now, for $t>0$, define the fibering maps $\phi_{u}:t\mapsto I_{\lambda}(tu)$ as

We note that

This gives that $tu\in \mathcal{N}_{\lambda}$ if and only if $\phi_{u}^{\prime}(t) = 0$ and in particular, $u\in \mathcal{N}_{\lambda}$ if and only if $\phi_{u}^{\prime}(1) = 0$. Moreover, for each $u\in \mathcal{N}_{\lambda}$, from (3.1), (3.3) and (3.5) we get that

which and (3.4) yield that $\mathcal{N}_{\lambda}^{+}$, $\mathcal{N}_{\lambda}^{-}$ and $\mathcal{N}_{\lambda}^{0}$ are corresponding to local minima, local maxima and points of inflection of $\phi_{u}(t)$, namely

In order to understand the Nehari manifold and the fibering maps, we consider the function $m_{u}:\mathbb{R}^{+}\to \mathbb{R}$ defined by

Clearly, for any $t>0$,

Thus, we obtain that

From the expression (3.5) of $\phi_{u}$, we see that the behavior of the fibering maps $\phi_{u}$ according to the sign of $\int_{\Omega} f(x)|u|^{\alpha} dx$ and $\int_{\Omega} h(x)|u|^{\beta} dx$, then we will study the following four cases.

Case 1: $\int_{\Omega} f(x)|u|^{\alpha} dx<0$ and $\int_{\Omega} h(x)|u|^{\beta} dx<0$. In this case, we have that $m_{u}(t)>0$ for all $t>0$, and the equation

has no solution for all $\lambda>0$. See Fig. 1.

Case 2: $\int_{\Omega} f(x)|u|^{\alpha} dx>0$ and $\int_{\Omega} h(x)|u|^{\beta} dx<0$. Since $\int_{\Omega} h(x)|u|^{\beta} dx<0$, we have $m_{u}(t)>0$ for all $t>0$, and $\lim\limits_{t\to +\infty}m_{u}(t)\to +\infty$. Moreover, for all $t>0$,

This gives that $m_{u}$ is strictly increasing on $(0,+\infty)$. Then, there exists a unique $t_1 = t_{1}(u)>0$ such that $m_u(t_1) = \lambda\int_{\Omega}f(x) |u|^{\alpha} dx$, see Fig. 2. Moreover, for all $t\in (0,t_{1})$, $m_u(t)<\lambda\int_{\Omega}f(x) |u|^{\alpha} dx$, and for all $t\in (t_{1},+\infty)$, $m_u(t)>\lambda\int_{\Omega}f(x) |u|^{\alpha} dx$. So, together with (3.6) we have that $\phi_u(t)$ is decreasing on $(0,t_1)$, increasing on $(t_1,\infty)$ and $\phi_{u}'(t_1) = 0$. Those imply that the function $\phi_{u}$ has exactly one critical point $t_{1}>0$ such that $t_{1}u \in \mathcal{N}^{+}_{\lambda}$, that is, $t_{1}>0$ is a global minimum point of function $\phi_{u}$, see Fig. 3.

Case 3: $\int_{\Omega} f(x)|u|^{\alpha} dx<0$ and $\int_{\Omega} h(x)|u|^{\beta} dx>0$. Since $\beta-\alpha>p-\alpha$ and $\int_{\Omega} h(x)|u|^{\beta} dx>0$, we have $m_{u}(t)\to -\infty$ as $t\to +\infty$. Moreover, from $m_{u}(0) = 0$ and $\lim_{t\to 0^{+}}m_{u}'(t)>0$, we get that the problem (3.8) has a unique solution $t_{2} = t_{2}(u)>0$ for each $u$, see Fig. 4. Similarly as Case 2, we obtain that $\phi_{u}'(t)>0$ for all $t\in (0,t_2)$, and $\phi_{u}'(t)<0$ for all $t\in (t_2,+\infty)$, which yields that $\phi_u$ is increasing on $(0,t_2)$, decreasing on $(t_2, \infty)$. So, $t_{2}>0$ is the global maximum point of $\phi_{u}$ such that $t_{2}u \in \mathcal{N}^{-}_{\lambda}$, see Fig. 5.

Case 4: $\int_{\Omega} f(x)|u|^{\alpha} dx>0$ and $\int_{\Omega} h(x)|u|^{\beta} dx>0$. Similarly as Case 3, we get that $m_{u}(0) = 0$, $m_{u}(t)\to -\infty$ as $t\to +\infty$ and there exists

such that $m_u$ is increasing on $(0,t_{\max})$, decreasing on $(t_{\max}, \infty)$ and the function $m_{u}$ attains its maximum value at $t_{\max}$. See Fig. 6. Moreover,

If $0<\lambda\int_{\Omega}f(x)|u|^{\alpha}dx<m_{u}(t_{\max})$, there exist $t_{3}: = t_{3}(u)$, $t_{4}: = t_{4}(u)>0$ such that $t_{3}<t_{\max}<t_{4}$ and

Then, $\phi_{u}'(t)<0$ for all $t\in (0,t_3)$, $\phi_{u}'(t)>0$ for all $t\in (t_3, t_{4})$, and $\phi_{u}'(t)<0$ for all $t\in (t_4,+\infty)$, that is, $\phi_u$ is decreasing on $(0,t_3)$, increasing on $(t_3, t_{4})$, and decreasing on $(t_4,+\infty)$. So, under the condition of $0<\lambda\int_{\Omega}f(x)|u|^{\alpha}dx<m_{u}(t_{\max})$, there exist $t_{3}<t_{\max}<t_{4}$ that $\phi_u$ has a local minimum point $t_3>0$ and a local maximum point $t_4>0$ with $t_3 u\in\mathcal{N}_{\lambda}^{+}$ and $t_4 u\in\mathcal{N}_{\lambda}^{-}$. See Fig. 7.

From the above proof of Case 4, we have the following result.

Lemma 3.3. For some $u\in E_{g}^{0}\backslash \{0\}$ with $\int_{\Omega} f(x)|u|^{\alpha} dx>0$ and $\int_{\Omega} h(x)|u|^{\beta} dx>0$. Then, there exists a positive constant $\lambda_{0}>0$ such that for any $0<\lambda<\lambda_0$, there are unique $\tau_{1}$, $\tau_{2}>0$ such that $\tau_{1}<t_{max}<\tau_{2}$ and $\tau_1 u\in\mathcal{N}_{\lambda}^{+}$ and $\tau_2 u\in\mathcal{N}_{\lambda}^{-}$. Moreover,

Proof. From the proof of Case 4, we only need to prove that there exists $\lambda_{0}>0$ such that, for any $\lambda\in (0,\lambda_{0})$, the following inequality holds

Indeed, since

Thus, taking $\lambda_{0}: = \left[\left(\frac{p-\alpha}{\beta-\alpha}\right)^{\frac{p-\alpha}{\beta-p}}- \left(\frac{p-\alpha}{\beta-\alpha}\right)^{\frac{\beta-\alpha}{\beta-p}}\right] \frac{S_{p^*_{s}}^{\frac{\beta-\alpha}{\beta-p}}}{\|f\|_{\frac{p^{*}_{s}}{p^{*}_{s}-\alpha}} \|h\|_{\frac{p^{*}_{s}}{p^{*}_{s}-\beta}}^{\frac{p-\alpha}{\beta-p}} }$, the inequality (3.9) holds for all $\lambda\in (0,\lambda_{0})$ and completes the proof of Lemma 3.3.

4.   Existence of solutions on $\mathcal{N}_{\lambda}^{+}$ and $\mathcal{N}_{\lambda}^{-}$

In this section, we use the results in Section 3 to prove the existence of a nontrivial solution on $\mathcal{N}_{\lambda}^{+}$, as well as on $\mathcal{N}_{\lambda}^{-}$. First, we state the following result. Let

Lemma 4.1. For each $0<\lambda<\Lambda_{0}$, we have $\mathcal{N}^{0}_{\lambda} = \emptyset$.

Proof. We consider the following two cases.

Case Ⅰ: $u\in \mathcal{N}_{\lambda}$ and $\int_{\Omega}h(x)|u|^{\beta}dx\leq 0$. We have

Thus, $\langle \Phi'_{\lambda}(u),u\rangle = (p-\alpha)\|u\|^{p}_{E^{0}_{g}}-(\beta-\alpha)\int_{\Omega}h(x)|u|^{\beta}dx>0$ and so $u\notin \mathcal{N}^{0}_{\lambda}$.

Case Ⅱ: $u\in \mathcal{N}_{\lambda}$ and $\int_{\Omega}h(x)|u|^{\beta}dx> 0$. Suppose that $\mathcal{N}^{0}_{\lambda}\neq\emptyset$ for all $\lambda\in(0,\Lambda_0)$. By (3.3), for each $u\in \mathcal{N}^{0}_{\lambda}$ we have

and

Thus, (4.1), (4.2) and the Sobolev inequality imply that

and

where $S_{p^*_{s}}>0$ is the best constant of Sobolev embedding. So, (4.3) and (4.4) imply that

contradicting with the assumption.

Let

By Lemmas 3.1, 3.2 and 4.1, for each $\lambda\in (0,\Lambda_{*})$, we get $\mathcal{N}_{\lambda}^{+}\cap \mathcal{N}_{\lambda}^{-} = \emptyset$, $\mathcal{N}_{\lambda}^{+}\cup \mathcal{N}_{\lambda}^{-} = \mathcal{N}_{\lambda}$, and the energy functional $I_{\lambda}$ is coercive and bounded from below on $\mathcal{N}_{\lambda}$, $\mathcal{N}_{\lambda}^{+}$ and $\mathcal{N}_{\lambda}^{-}$. Then, the Ekeland variational principle implies that $I_{\lambda}$ has a minimizing sequence on each manifold of $\mathcal{N}_{\lambda}$, $\mathcal{N}_{\lambda}^{+}$ and $\mathcal{N}_{\lambda}^{-}$. Define

Lemma 4.2. The following facts hold.

(ⅰ) If $\lambda\in(0,\Lambda_{*})$, then we have $c_{\lambda}\leq c^{+}_{\lambda}<0$.

(ⅱ) If $\lambda\in(0,\Lambda_{*})$,then we have $c^{-}_{\lambda}>c_{0}$ for some positive constant $c_{0}$ depending on $\lambda, p,s,\alpha,\beta$ and $S_{p^{*}_{s}}$.

Proof. (ⅰ) Let $u\in \mathcal{N}^{+}_{\lambda}$, by (3.3) we have

Hence

From this inequality and the definition of $c_{\lambda}$, $c^{+}_{\lambda}$, we deduce that $c_{\lambda}\leq c^{+}_{\lambda}<0$.

(ⅱ) Let $u\in \mathcal{N}^{-}_{\lambda}$, from (3.3) we have

Moreover, by the Hölder inequality and the Sobolev embedding theorem, we obtain

which and (4.5) imply that

Putting together (3.2) and (4.6), we have

Thus, if $\lambda\in(0,\Lambda_{*})$, we get that $I_{\lambda}(u)>c_{0}>0$ for all $u\in\mathcal{N}^{-}_{\lambda}$ for some positive constant $c_{0} = c_{0}(\lambda,p,s,\alpha,\beta,S_{p^*_{s}})$, and completes the proof.

Theorem 4.1. Let $\mathbb{G}$ be a homogeneous Lie group with homogeneous dimension $Q$. Assume that $p > 1$, $s\in (0, 1)$, $Q > sp$ and $\lambda\in (0,\Lambda_*)$ hold. Then $I_{\lambda}$ has a local minimizer $u^{+}_{\lambda}$ in $\mathcal{N}_{\lambda}^{+}$ satisfying $I_{\lambda}(u^{+}_{\lambda}) = c^{+}_{\lambda} = c_{\lambda}$ and $u^{+}_{\lambda}$ is a nontrivial nonnegative solution of (1.1).

Proof. Since $I_{\lambda}$ is coercive and bounded from below on $\mathcal{N}_{\lambda}$ and so on $\mathcal{N}_{\lambda}^{+}$, by the Ekeland variational principle, there exists a minimizing sequence $\{u_k\}_{k}\subset\mathcal{N}_{\lambda}^{+}$ such that

From Lemma 3.3 and Case 4, we known that $c_{\lambda}^{+}<0$.

As $I_{\lambda}$ is coercive on $\mathcal{N}_{\lambda}$, $\{u_k\}_{k}$ is a bounded sequence in $E^{0}_{g}$. Therefore, by Theorem 2.2, there is a subsequence, still denoted by $\{u_k\}_{k}$, and $u_\lambda^{+} \in E_{g}^{0}$ such that

By the Hölder inequality and Dominated convergence theorem and (4.8), we obtain

and

First, we claim that $\int_{\Omega}f(x)|u_{\lambda}^{+}|^{\alpha}dx\neq 0$, we argue by contradiction, then we have $\int_{\Omega}f(x)|u_{k}|^{\alpha}dx\to 0$ as $k\to \infty$. Thus

and

This contradicts $I_{\lambda}(u_{k})\to c_{\lambda}<0$ as $k\to \infty$.

From (4.8), (4.9) and (4.10), we know $u_{\lambda}^{+}$ is a weak solution of (1.1). We now claim $u_{\lambda}^{+}$ is a nontrivial solution of (1.1). Since $u_{k}\in \mathcal{N}_{\lambda}$, we have

which implies that

Let $k\rightarrow \infty$ in (4.12), by (4.7), (4.8) and $c_{\lambda}^{+}<0$, we have that

Using this inequality, we get that $u_{\lambda}^{+}$ is a nontrivial solution of (1.1).

Next, we show that $u_{k}\to u_{\lambda}^{+}$ in $E_{g}^{0}$ and $I_{\lambda}(u_{\lambda}^{+}) = c_{\lambda}^{+}$. Since $u_{\lambda}^{+}\in \mathcal{N}_{\lambda}$ and (4.11), we obtain

Hence, thanks to (4.13) we obtain $I_{\lambda}(u_{\lambda}^{+}) = c_{\lambda}^{+}$ and $\lim\limits_{k\to\infty}\|u_{k}\|^{p}_{E_{g}^{0}} = \|u_{\lambda}^{+}\|^{p}_{E_{g}^{0}}$, this implies that $u_{k}\to u_{\lambda}^{+}$ in $E_{g}^{0}$.

Finally, we claim that $u_{\lambda}^{+}\in \mathcal{N}^{+}_{\lambda}$. Assume by contradiction that $u_{\lambda}^{+}\in \mathcal{N}^{-}_{\lambda}$, then by Lemma 3.3, there exist unique $\tau_{\lambda}^{+}$ and $\tau_{\lambda}^{-}>0$ such that $\tau_{\lambda}^{+}u_{\lambda}^{+}\in \mathcal{N}^{+}_{\lambda}$ and $\tau_{\lambda}^{-}u_{\lambda}^{+}\in \mathcal{N}^{-}_{\lambda}$. In particular, we have $\tau_{\lambda}^{+}<\tau_{\lambda}^{-} = 1$. Since

there exists $t_{*}\in (\tau_{\lambda}^{+},\tau_{\lambda}^{-})$ such that $I_{\lambda}(\tau_{\lambda}^{+}u_{\lambda}^{+})<I_{\lambda}(t_{*}u_{\lambda}^{+})$, from which and Lemma 3.3, we have

a contraction. Note that $I_{\lambda}(u_{\lambda}^{+}) = I_{\lambda}(|u_{\lambda}^{+}|)$ and $|u_{\lambda}^{+}|\in \mathcal{N}_{\lambda}$, so by Lemma 3.2, we obtain that $u_{\lambda}^{+}$ is a nontrivial nonnegative solution of (1.1) and conclude the proof.

Next, we establish the existence of a local minimum for $I_{\lambda}$ on $\mathcal{N}^{-}_{\lambda}$.

Theorem 4.2. Let $\mathbb{G}$ be a homogeneous Lie group with homogeneous dimension $Q$, and let $p > 1$, $s\in (0, 1)$, $Q > sp$ and $\lambda\in (0,\Lambda_*)$. Then $I_{\lambda}$ has a local minimizer $u_{\lambda}^{-}$ in $\mathcal{N}^{-}_{\lambda}$ satisfying $I_{\lambda}(u^{-}_{\lambda}) = c^{-}_{\lambda}$ and $u^{-}_{\lambda}$ is a nontrivial nonnegative solution of (1.1).

Proof. From Lemma 4.2, we get $I_{\lambda}(u)\geq c_{0}>0$ for any $u\in \mathcal{N}^{-}_{\lambda}$. Hence $c_{\lambda}^{-}\geq c_{0}>0$ and there is a minimizing sequence $\{u_k\}_{k}\subset \mathcal{N}_{\lambda}^{-}$ such that

Hence, (4.14) and the coerciveness of $I_{\lambda}$ yield that $\{u_k\}_{k}$ is a bounded in $E_{g}^{0}$. Without loss of generality, we can suppose that $u^{-}_{\lambda}\in \mathcal{N}^{-}_{\lambda}$ such that $u_{k}\rightharpoonup u^{-}_{\lambda}$ in $E_{g}^{0}$, and $u_{k}\to u^{-}_{\lambda}$ in $L^{q}(\Omega)$ for any $q\in [1,p^{*}_{s})$. Using this results, as the proof of Theorem 4.1, we get that

and

Now we claim that $\int_{\Omega} f(x)|u^{-}_{\lambda}|^{\alpha}dx\neq0$. To see this, by contradiction, we suppose that $\int_{\Omega} f(x)|u_{k}|^{\alpha}dx\to0$ as $k\to\infty$, from which we have

and

This contradicts $I_{\lambda}(u_{k})\to c_{\lambda}<0$ as $k\to\infty$.

Finally we prove $u_{k}\rightarrow u^{-}_{\lambda}$ in $E_{g}^{0}$. Other case, we have

Which contradicts $u^{-}_{\lambda}\in \mathcal{N}^{-}_{\lambda}$. Hence $u_{k}\to u^{-}_{\lambda}$ in $E_{g}^{0}$ as $k\to \infty$. Similar to Theorem 4.1, the proof can be completed.

Now, we complete the proof of Theorem 1.1.

Proof of Theorem 1.1. For $0<\lambda<\Lambda_{*}$, by Theorems 4.1 and 4.2, there are $u^{+}_{\lambda}\in \mathcal{N}_{\lambda}^{+}$ and $u^{-}_{\lambda}\in \mathcal{N}_{\lambda}^{-}$ such that

So, the equation (1.1) admits at least two nontrivial nonnegative solutions $u^{+}_{\lambda}$ and $u^{-}_{\lambda}$. Since $\mathcal{N}^{+}_{\lambda}\cap\mathcal{N}^{-}_{\lambda} = \emptyset$, it results $u^{+}_{\lambda}$ and $u^{-}_{\lambda}$ are distinct nontrivial nonnegative solutions of problem (1.1) and the thesis is proved.