Strongly-coupled and predator-prey subelliptic system on the Heisenberg group

1.   Introduction

The spatial distribution pattern of an animal population in its natural environment may be the result of several biological effects. In a patchy environment, linear diffusional flows have a stabilizing effect on the coexistence of competitive species. Shigesada, Kawasaki, and Teramoto ^[22] studied the spatial segregation of interacting species and proposed the model

where $u_1$ and $u_2$ represent the densities of two competing species, $d_1$ and $d_2$ are their diffusion rates, $a_1$ and $a_2$ denote the intrinsic growth rates, $b_1$ and $b_2$ account for intra-specific competitions, $c_1$ and $c_2$ are the coefficients of inter-specific competitions, $\alpha_{11}$ and $\alpha_{22}$ are usually referred as self-diffusion pressures, and $\alpha_{12}$ and $\alpha_{21}$ are cross-diffusion pressures. Here, $\Delta$ is the Laplace operator, $\Omega$ is a bounded smooth domain of $R^N$ with $N\geq 1$, $\partial\Omega$ and $\overline{\Omega}$ are the boundary and the closure of $\Omega$, respectively, $\Omega_T = \Omega\times[0, T)$ and $\partial\Omega_T = \partial\Omega\times[0, T)$ for some $T \in (0, \infty]$, $\nu$ is the outward unit normal vector on $\partial\Omega$, $d_i, a_i, b_i, c_i(i = 1, 2)$ are all positive constants, and $\alpha_{ij} (i, j = 1, 2)$ denote non-negative constants. The initial values $u_{1, 0}$ and $u_{2, 0}$ are non-negative smooth functions that are not identically zero. For more details on the backgrounds of this model, we refer to ^[21,22]; for reaction diffusion, see ^[7,15].

Lou and Ni ^[16] considered positive steady-state solutions to the above strongly-coupled parabolic system and derived properties of these solutions, including a priori estimates, as well as conditions for existence and non-existence. To prove those results, they first considered the strongly-coupled elliptic system

For $N = 1, \alpha_{11} = \alpha_{21} = \alpha_{22} = 0$, Mimura and Kawasaki ^[19] demonstrated the existence of small amplitude solutions bifurcating from the trivial solution. Mimura ^[18] established that large amplitude solutions exist when $\alpha_{12}$ is suitably large. Mimura, Nishiura, Tesei, and Tsujikawa ^[20] proved the existence of non-constant solutions of this problem. Jia and Xue ^[14] investigated the non-existence of non-constant positive steady states in a generalized predator-prey system. Xue, Jia, Ren, and Li ^[28] proved both the existence and non-existence of non-constant positive stationary solutions for the general Gause-type predator-prey system with constant self-diffusion and cross-diffusion. For more information on the parabolic system, we refer to ^[21,27,29].

In this paper, we study the strongly-coupled subelliptic system on the Heisenberg group

where $\Delta_{\mathbb{H}}$ is the degenerate subelliptic (also called hypoelliptic in ^[12]) operator. Here, $d_i, a_i, b_i, c_i(i = 1, 2)$ are positive constants, and $\alpha_{ij} (i, j = 1, 2)$ are non-negative constants. For the degenerate of the $\Delta_{\mathbb{H}}$, there are some different forms ^[14,16,28]; see Section 2 for further details.

Only one of the diffusion rates or one of the self-diffusion pressures needs to be large to prevent the formation of a non-constant solution to (1.1).

Theorem 1.1. Suppose that $\frac{a_1}{a_2}\neq \frac{b_1}{b_2}$ and $\frac{a_1}{a_2}\neq \frac{c_1}{c_2}$.

(i) There exists a positive constant $C_1 = C_1(d_i, a_i, b_i, c_i, \alpha_{12}, \alpha_{21})$ such that problem (1.1) has no non-constant solution if $\max\{\alpha_{11}, \alpha_{22}\}\geq C_1$.

(ii) There exists a positive constant $C_2 = C_2(a_i, b_i, c_i, \alpha_{ij})$ such that if $\ max\{d_{1}, d_{2}\}\geq C_2$, then problem (1.1) has no non-constant solution provided that both $\alpha_{11}$ and $\alpha_{22}$ are positive.

In the case of weak competition, if self-diffusion is weaker than diffusion, then (1.1) still has no non-constant solution.

To obtain some non-existence results from Theorem 1.1, we mainly study the effects of diffusion and self-diffusion in the strongly-coupled subelliptic system

For the sake of convenience, we collect here all the assumptions on $f, g$, some of which will be made at different times in this paper. Throughout this paper, we always follow the following hypotheses:

(H1) $f(0, 0) = a_1, g(0, 0) = a_2, \frac{\partial f}{\partial u}\leq-b_1, \frac{\partial g}{\partial u}\leq-b_2, \frac{\partial f}{\partial v}\leq-c_1, \frac{\partial g}{\partial v}\leq-c_2,$ for all $u\geq0, v\geq0,$ where $a_i, b_i$ and $c_i$ are all positive constants for $i = 1, 2.$

(H1') $f(0, 0) = a_1, g(0, 0) = a_2, \frac{\partial f}{\partial u}\leq-b_1, \frac{\partial g}{\partial u}\leq0, \frac{\partial f}{\partial v}\leq0, \frac{\partial g}{\partial v}\leq-c_2,$ for all $u\geq0, v\geq0,$ where $a_1, a_2, b_1$ and $c_2$ are all positive constants.

(H2) Both $\{u > 0\; |\; f(u, 0) = g(u, 0) = 0\}$ and $\{v > 0\; |\; f(0, v) = g(0, v) = 0\}$ are empty.

(H3) $f(u, v) = g(u, v) = 0$ has a unique positive root $(u^*, v^*)$.

It is easy to see that (H1) is more restrictive than (H1′). From (H1′), it follows that if $(u, v)$ is a positive root of $f(u, v) = g(u, v) = 0$, then $u\leq \frac{a_1}{b_1}$ and $v\leq \frac{a_2}{c_2}$. For the special case $f = u(a_1-b_1u-c_1v)$ and $g = v(a_2-b_2u-c_2v)$, it is trivial to check that (H1) and (H1′) hold, and that (H2) is equivalent to $\frac{a_1}{a_2}\neq \frac{b_1}{b_2}$ and $\frac{a_1}{a_2}\neq \frac{c_1}{c_2}$, while (H3) is satisfied only in $\frac{b_1}{b_2} > \frac{a_1}{a_2} > \frac{c_1}{c_2}$ and $\frac{b_1}{b_2} < \frac{a_1}{a_2} < \frac{c_1}{c_2}$.

For the generalized predator-prey subelliptic system with cross-diffusion and homogeneous Neumann boundary conditions, we investigate the existence and non-existence of non-constant positive solutions to the following subelliptic system

The functions $q(u)\in C^1([0, +\infty))$ and $p(u)\in C^1([0, +\infty))\bigcap C^2((0, +\infty))$ are assumed to satisfy the following two hypotheses throughout this paper:

(H4) $q(0) > 0, q'(u) < 0,$ for all $u\geq0$. And there exists a unique positive constant $S$, such that $q(S) = 0$.

(H5) $p(0) = 0, \lim\limits_{u\rightarrow 0^+}p'(u) < \infty, cp(S) > a_2$ and $p'(u) > 0$ for all $0 < u\leq S$.

For the case $\bar{\alpha}_{21} = 0$ of the subelliptic system (1.3), we have the following theorem.

Theorem 1.2. There are positive constants $\tilde{d}_1, \tilde{d}_2, \tilde{\alpha}_{12}$ such that if $d_1\geq \tilde{d}_1, d_2\geq \tilde{d}_2$ and $\bar{\alpha}_{12}\leq \tilde{\alpha}_{12}$, then problem (1.3) has no non-constant solution.

In general, for the subelliptic system (1.3), we obtain the following theorem.

Theorem 1.3. There are positive constants $\tilde{d}_1, \tilde{d}_2, \tilde{\alpha}_{12}, \tilde{\alpha}_{21}$ such that if $d_1 > \tilde{d}_1, d_2 > \tilde{d}_2$ and $\bar{\alpha}_{12} < \tilde{\alpha}_{12}, \bar{\alpha}_{21} < \tilde{\alpha}_{21}$, then problem (1.3) has no non-constant solution.

The paper is organized as follows. In Section 2, we collect some well-known facts about ${\mathbb{H}}^{n}$ and the subelliptic operator ${{\Delta }_{\mathbb{H}}}$. Section 3 gives an overview of competition-diffusion in the strongly-coupled model. Section 4 is devoted to studying diffusion and self-diffusion in the strongly-coupled model. In Section 5, we derive the predator-prey model.

2.   Preliminaries

In this section, we list some facts related to the Heisenberg group and sub-Laplacian ${{\Delta }_{\mathbb{H}}}$. For proofs and more information, we refer, for example, to ^[3,4,8,9,12].

The Heisenberg group ${\mathbb{H}}^{n}$ is the Euclidean space ${{\mathbb{R}}^{2n+1}}(n\ge 1)$ endowed with the group action $\circ$ defined by

where $\xi = ({{x}_{1}}, \cdots, {{x}_{n}}, {{y}_{1}}, \cdots, {{y}_{n}}, t): = (x, y, t)\in {\mathbb{R}}^{n}\times{\mathbb{R}}^{n}\times{\mathbb{R}}$, ${{\xi }_{0}} = ({{x}_{0}}, {{y}_{0}}, {{t}_{0}})$. Let us denote by $\delta_{\lambda}$ the dilations on ${{\mathbb{R}}^{2n+1}}$, i.e.,

which satisfies ${{\delta }_{\lambda }}({{\xi }_{0}}\circ \xi) = {{\delta }_{\lambda }}({{\xi }_{0}})\circ {{\delta }_{\lambda }}(\xi)$.

The left invariant vector fields corresponding to ${\mathbb{H}}^{n}$ are of the form

The Heisenberg gradient of a function $u$ is defined as

The sub-Laplacian ${{\Delta }_{\mathbb{H}}}$ on ${\mathbb{H}}^{n}$ is

with the expansion

It is easy to check that

and $\left\{ {{X}_{1}}, \cdots, {{X}_{n}}, {{Y}_{1}}, \cdots {{Y}_{n}} \right\}$ satisfies the Hörmander's rank condition (see ^[12]). In particular, this implies that ${{\Delta }_{\mathbb{H}}}$ is hypoelliptic (see ^[12]), and the solution of equation including ${{\Delta }_{\mathbb{H}}}$ satisfies the maximum principle (see ^[2,4]).

Denote by $Q = 2n+2$ the homogeneous dimension of ${\mathbb{H}}^{n}$. The norm ${{\left| \xi \right|}_{\mathbb{H}}}$ is the distance of $\xi$ to the origin (see ^[8]),

Using this norm, one can define the distance between two points in ${\mathbb{H}}^{n}$ in the natural way

where $\eta^{-1}$ denotes the inverse of $\eta$ with respect to the group action $\circ$, i.e., $\eta^{-1} = -\eta$.

The open ball of radius $R > 0$ centered at $\xi_{0}$ is the set

By the dilation of the group, $\xi \to {{\left| \xi \right|}_{\mathbb{H}}}$ is homogeneous of degree one with respect to $\delta_{\lambda}$ and

where $\left| \cdot \right|$ denotes the Lebesgue measure. Noting that $X_{i}$ and $Y_{i}$ are homogeneous of degree minus one with respect to $\delta_{\lambda}$, i.e.,

then ${{\Delta }_{\mathbb{H}}}$ is homogeneous of degree minus two and left invariant.

We denote the Sobolev space by

and

which is a Banach space about the norm

Denote by $W_{0}^{1, 2}(\Omega)$ the closure of $C_{0}^{\infty }(\Omega)$ in $W^{1, 2}(\Omega)$.

Let us state Sobolev's and Poincaré's inequalities in ${{\mathbb{H}}^{n}}$, see ^[10,13,17].

Lemma 2.1. Let $U$ be a bounded domain in ${{\mathbb{H}}^{n}}$ and $\Omega\subset\subset U$. If $1 < p < Q$ and $u\in W^{1, p}_{0}(\Omega)$, then there exists $C > 0$ depending on $n, p$ and $\Omega$, such that for any $1\le q\le \frac{pQ}{Q-p}$,

If $1\leq p < \infty$ and $u\in W^{1, p}_{0}(\Omega)$, then

For the maximum principle, we refer to ^[2,4].

Lemma 2.2. Let $\Omega$ be a bounded domain and $K(\xi) > 0$, $u$ satisfies

then $u\geq0$ on $\Omega$. Furthermore, $u > 0$ on $\Omega$, unless $u\equiv0$.

The following Hopf-type lemma is from ^[4,6,26].

Lemma 2.3. For a domain $V$ in $\hat{\mathbb{H}}^{n}: = {{\mathbb{H}}^{n}}\times {\mathbb{R}}\subset {{\mathbb{C}}^{n+1}}$, let the point ${{P}_{0}}\in \partial V$ satisfy the interior Heisenberg ball condition (see ^[6]). Assume that $U\in {{C}^{2}}(V)\cap {{C}^{1}}(\overline{V})$ is a solution to

for ${{c}_{1}(z)}\in {{L}^{\infty }}(V)$, where

If $U(z) > U({{P}_{0}}) = 0$, $z\in V$, then

where $\nu$ is the outer unit normal to $\partial V$ at ${{P}_{0}}$.

If ${{c}_{1}(z)} = 0$, then the above conclusion is also valid when we drop the assumption $U({{P}_{0}}) = 0$.

To handle the equations in this paper, we also give a maximum principle as follows.

Lemma 2.4. Suppose that $h\in C(\overline{\Omega}\times \mathbb{R})$.

(i) If $w\in C^2(\Omega)\bigcap C^1(\overline{\Omega})$ satisfies

and $w(\xi_0) = \max\limits_{\overline{\Omega}} w$, then $h(\xi_0, w(\xi_0))\geq0$.

(ii) If $w\in C^2(\Omega)\bigcap C^1(\overline{\Omega})$ satisfies

and $w(\xi_0) = \min\limits_{\overline{\Omega}} w$, then $h(\xi_0, w(\xi_0))\leq0$.

Proof. We prove (ⅰ) only since (ⅱ) can be established in a similar way.

If $\xi_0\in{\Omega}$. Since $w(\xi_0) = \max\limits_{\overline{\Omega}} w$, we have ${{\Delta }_{\mathbb{H}}}w(\xi_0)\leq 0$. Thus, the conclusion holds from (2.8).

If $\xi_0\in{\partial\Omega}$. We argue by contradiction. Suppose that $h(\xi_0, w(\xi_0)) < 0$. Then, by the continuity of $h$ and $w$, there exists a small ball $B_{\mathbb{H}}$ in $\overline{\Omega}$ with $\partial B_{\mathbb{H}}\bigcap \partial\Omega = \{\xi_0\}$ such that $h(\xi, w(\xi)) < 0$ for $\xi\in B_{\mathbb{H}}$. Therefore, by (2.8), we have ${{\Delta }_{\mathbb{H}}}w(\xi) > 0$ for all $\xi\in B_{\mathbb{H}}$.

Since $w(\xi_0) = \max\limits_{\overline{B_{\mathbb{H}}}} w$, it follows from the Hopf boundary Lemma 2.3 that $\frac{\partial w}{\partial \nu }({{\xi}_{0}}) > 0$, which contradicts the boundary condition in (2.8). □

For the following Harnack inequality, we refer to ^[3,23,25].

Lemma 2.5. Let $\Omega$ be a bounded domain and $K(\xi)\in C(\overline{\Omega})$, $u$ satisfies

then there exists a positive constant $C = C(\| K(\xi) \|_{L^\infty (\Omega)}, \Omega)$, such that $\max\limits_{\overline{\Omega}}u\leq C\min\limits_{\overline{\Omega}}u$.

3.   Competition-diffusion model

In this section, we consider the non-existence of non-constant solutions to the following semilinear subelliptic system

Throughout this section, $C$ and $C_i$ will always denote generic positive constants depending only on $f, g$ and/or $\Omega$. Let $(u^*, v^*)$ be a positive root to $f(u, v) = g(u, v) = 0$,

and $|\mathcal{M}|$ denotes the determinant of the matrix $\mathcal{M}$.

Theorem 3.1. Suppose that (H1′) and (H3) hold. Then $(u, v) = (u^*, v^*)$ is the only solution of problem (3.1) if either

(i) $|\mathcal{M}| > 0$ or

(ii) $|\mathcal{M}|\leq 0$ and $\max \{d_1, d_2\}\geq C_1$ for some constant $C_1$.

To prove Theorem 3.1, we need some preliminary results. In this section, set

Lemma 3.2. Suppose that (H1′) and (H3) hold.

(i) If $|\mathcal{M}| > 0$, then

(ii) $|\mathcal{M}| < 0$, then

(iii) $|\mathcal{M}| = 0$, then there are three possibilities: the two sets $I_1$ and $I_2$ satisfy (3.3), or they satisfy (3.4), or one of them is equal to the set $\{(u^*, v^*)\}$.

Proof. We shall show (ⅰ) only, since parts (ⅱ) and (ⅲ) can be shown in similar ways. By (H1′), the curves $\Gamma_1, \Gamma_2$ can be represented as

It is easy to show that $F, G$ are non-increasing functions with $F(v^*) = u^*$ and $G(u^*) = v^*$. Then, our conclusion follows from (H3) and the observation that if $|\mathcal{M}| > 0$, $\Gamma_1$ lies above $\Gamma_2$ for $0 < u < u^*$ in $uv$ plane, and $\Gamma_1$ is below $\Gamma_2$ for $u\geq u^*$. □

Lemma 3.3. Suppose that (H1′) and (H3) hold.

(i) If $|\mathcal{M}| > 0$, then $(u, v) = (u^*, v^*)$ is the only solution of problem (3.1).

(ii) If $|\mathcal{M}|\leq 0$, then any solution $(u, v)$ of problem (3.1) satisfies the following estimate:

Proof. Let $u(\xi_0) = \max\limits_{\overline{\Omega}} u$, by Lemma 2.4 and (H1′), we have

and in a similar way, we can obtain that

The (3.6) and (3.7) show that

By Lemma 3.2, we have, if $|\mathcal{M}| > 0$,

This implies that $(u, v) = (u^*, v^*)$, hence (i) is established. Part (ii) follows similarly from Lemma 3.2. □

We shall present a priori estimates on solutions of the strongly-coupled subelliptic system (1.2).

Lemma 3.4. Suppose that (H1) holds. Then, there exists a positive constant $\widetilde{C} = \widetilde{C}(a_i, b_i, c_i)$ such that for any solution $(u, v)$ of problem (1.2) satisfying the following estimates:

Proof. Let $\Psi = u(d_1+\alpha_{11}u+\alpha_{12}v)$, then $\Psi$ satisfies

Let $\Psi(\xi_0) = \max\limits_{\overline{\Omega}}\Psi$, then by Lemma 2.2 and the positivity of $u$, we have $f(u(\xi_0), v(\xi_0))\geq 0$. Therefore,

where the last inequality follows from the assumption (H1) and $\eta_1 \geq 0, \eta_2 \geq 0$. Hence, we have $u(\xi_0)\leq \frac{a_1}{b_1}$ and $v(\xi_0)\leq \frac{a_1}{c_1}$. Then,

which in turn implies that

If $\alpha_{11}\leq d_1$, it follows directly from (3.9) that

If $\alpha_{11}\geq d_1$, by (3.9) we obtain

then

Combining (3.10) and (3.11), we obtain the first half of (3.8). The estimate of $\max\limits_{\overline{\Omega}} v$ can be obtained in a similar way. □

Proof of Theorem 3.1. By Lemma 3.3, it suffices to consider the case $|\mathcal{M}|\leq 0$. Let $(u, v)$ be an arbitrary solution of (3.1). We claim that there exists a positive constant $C$, independent of $(u, v)$, such that

where $\bar{u}$ is the average of $u$ in $\Omega$, i.e., $\bar{u} = \frac{1}{|\Omega|}\int_{\Omega}u$.

Following the proof of Lemma 3.4, by (H1′), we obtain

Substituting $u-\bar{u}$ into the problem (3.1), we have

where $\widetilde{f} = uf(u, v)$ can be estimated by

Multiplying (3.14) by $u-\bar{u}$, by Green's identity, Hölder's inequality, and Poincaré's inequality, we derive

which implies that

By Lemma 2.1 and (3.14), (3.15), we get

and hence, by Sobolev embedding theorem ^[3,11],

Since $\frac{2Q}{Q-4} > 2$, this proves (3.16). Iterating this argument finitely many times, we establish (3.12). Furthermore, it follows from (3.12) that

Then, we will show that there exists a positive constant $C$ (independent of $u$ and $v$), such that

It follows from the above process and Lemma 3.3 that

that is

which in turn implies that

Simultaneously, there exists a positive constant $C$ (independent of $u$ and $v$) such that the following inequality holds

From (3.7), it follows that for some $\zeta_1 > 0, \zeta_2 > 0,$

Hence, by (H1′) and (3.17), we deduce that

which, together with Lemma 3.3, shows that (3.19) holds.

At last, we prove that there exists a constant $C_1$ (independent of $u$ and $v$), such that if $\max\{d_1, d_2\}\geq C_1$, then the only solution of (3.1) is $(u, v) = (u^*, v^*)$.

Multiplying the first equation of (3.1) with $u-\bar{u}$, by Green's identity, we obtain

For the second equation of (3.1), we proceed slightly differently, as follows.

where $\varsigma_1(\xi)$ lies between $\bar{u}$ and $u(\xi)$, and $\varsigma_2(\xi)$ lies between $\bar{v}$ and $v(\xi)$ for each $\xi\in\Omega$. From the above conclusions, it follows that (3.18) and (3.19) hold. And then, by (3.18) and (3.19), if $d_1\geq C$,

Choosing $\varepsilon = \frac{c_2 v^*}{4}$ in (3.21), we have

Combing (3.20) and (3.22), we arrive at

which implies that if $d_1 > C_2$, then ${\nabla }_{\mathbb{H}}{u}\equiv0$, i.e., $u$ is constant.

Then, (3.22) guarantees that $v\equiv\bar{v}$, a non-negative constant.

In view of part (ii) of Lemma 3.3, we see that these constants must be positive. Hence, from the assumption (H3), we conclude that $(u, v)\equiv(u^*, v^*)$.

A similar argument applies when $d_2$ is large, leading to the same conclusion. This completes the proof of Theorem 3.1. □

As a consequence of Theorem 3.1, we have the following corollary:

Corollary 3.5. If $f = u(a_1-b_1u-c_1v)$ and $g = v(a_2-b_2u-c_2v)$, then $(u, v) = (u^*, v^*)$ is the only solution of problem (3.1) if either

(i) $\frac{b_1}{b_2} > \frac{a_1}{a_2} > \frac{c_1}{c_2}$ or

(ii) $\frac{b_1}{b_2} < \frac{a_1}{a_2} < \frac{c_1}{c_2}$ and $\max \{d_1, d_2\}\geq C_1$ for some constant $C_1$.

Remark 1. The equations in Theorem 3.1 and Corollary 3.5 involve subelliptic operators, which are more general than elliptic operator as described in ^[16], and the proof mainly relies on Lemma 2.4, which is the subelliptic case.

4.   Diffusion and self-diffusion model

In this section, we mainly study the effects of diffusion and self-diffusion in the strongly-coupled subelliptic system (1.2). Throughout this section, $C$ will always denote generic positive constants depending only on $d_1, d_2, \alpha_{12}, \alpha_{21}$ and the nonlinearity $f, g$, but independent of $\alpha_{11}, \alpha_{22}$.

Theorem 4.1. Suppose that the conditions (H1) and (H2) hold. Then, there exists a constant $C$ such that if $\max\{\alpha_{11}, \alpha_{22}\}\geq C$, the problem (1.2) has no non-constant solution.

Lemma 4.2. Suppose that (H1) and (H2) hold.

(i) If $f(u, v) = g(u, v) = 0$ has no positive root, then there exists a constant $C$ such that (1.2) has no solution provided that $\max\{\alpha_{11}, \alpha_{22}\}\geq C$.

(ii) If $f(u, v) = g(u, v) = 0$ has at least a positive root, then there every small $\varepsilon > 0$, there exists a constant $C(\varepsilon)$ such that if $\max\{\alpha_{11}, \alpha_{22}\}\geq C(\varepsilon)$, for any solution $(u, v)$ of (1.2), there are two positive constants $\hat{u}, \hat{v}$ that $f(\hat{u}, \hat{v}) = g(\hat{u}, \hat{v}) = 0$ and $\| u-\hat{u} \|_{L^\infty (\Omega)}+\| v-\hat{v} \|_{L^\infty (\Omega)}\leq \varepsilon$.

Proof. We prove (ⅱ) at first; suppose that the conclusion is false. Without loss of generality, we assume that there exists a constant $\varepsilon_0 > 0$, and a sequence $\{\alpha_{11, k}, \alpha_{22, k}\}_{k = 1}^{\infty}$ with $\alpha_{11, k}\rightarrow {\infty}$, such that

for any positive root $(\hat{u}, \hat{v})$ of $f(u, v) = g(u, v) = 0$, where $(u_k, v_k)$ is a solution to

We use the same notation of the subsequence of $\{u_{k}\}_{k = 1}^{\infty}$ as for the original sequence $\{u_{k}\}_{k = 1}^{\infty}$, such that $u_{k}$ converges uniformly to a constant as $k\rightarrow {\infty}$. Set

then $\Phi_k$ satisfies

By Lemma 3.4 and the fact $\alpha_{11, k}\rightarrow {\infty}$, we know that $\| \Phi_k \|_{L^\infty (\Omega)}\leq C$. Hence by standard $L^p$ estimates and the Sobolev embedding theorem ^[5,11,24], we obtain $\| \Phi_k \|_{C^{1, \alpha} (\overline{\Omega})}\leq C$ for some $\alpha \in (0, 1)$. Therefore, a subsequence of $\{\Phi_{k}\}_{k = 1}^{\infty}$ converges to some nonnegative function $\Phi$ in $C^{1} (\overline{\Omega})$, and $\Phi$ must satisfy the following problem weakly

By standard subelliptic regularity theory, $\Phi\in C^{2} (\overline{\Omega})$ and therefore $\Phi = \hat{\Phi}$, where $\hat{\Phi}$ is a nonnegative constant. Letting $\hat{u} = \sqrt{\hat{\Phi}}$, we get that

as $k\rightarrow \infty$. Hence $u_k\rightarrow \hat{u}$ uniformly.

Next, we claim the subsequence of $\{v_{k}\}_{k = 1}^{\infty}$ and also denote $\{v_{k}\}_{k = 1}^{\infty}$, such that $v_{k}\rightarrow \hat{v}$ uniformly as $k\rightarrow {\infty}$, where $\hat{v}$ is some nonnegative constant.

Before establishing the above assertion, we show how to derive a contradiction via the fact that $(u_k, v_k)\rightarrow (\hat{u}, \hat{v})$ uniformly as $k\rightarrow {\infty}$.

Integrating the equations of (4.2) in $\Omega$, we have

From this, we conclude that $f(\hat{u}, \hat{v}) = g(\hat{u}, \hat{v}) = 0$ for $(\hat{u}, \hat{v})$. Suppose that $f(\hat{u}, \hat{v})\neq0$. Without loss of generality, we may assume that $f(\hat{u}, \hat{v}) > 0$. Since $(u_k, v_k)\rightarrow (\hat{u}, \hat{v})$ uniformly, $f(u_k, v_k)\rightarrow f(\hat{u}, \hat{v})$ as $k\rightarrow {\infty}$. Hence, $f(\hat{u}_k, \hat{v}_k) > 0$ for $k$ large, and therefore

for large $k$ since $u_k$ is always positive, which contradicts (4.3). A similar contradiction can be deduced if $g(\hat{u}, \hat{v})\neq0$.

By (H2) and the assumption that $f(0, 0) = a_1 > 0$, $g(0, 0) = a_2 > 0$, we must have

That is, $(u_k, v_k)\rightarrow (\hat{u}, \hat{v})$ uniformly with

which contradicts (4.1) and thus establishes (ⅱ) of Lemma 4.2.

To finish the proof of part (ⅱ) of Lemma 4.2, it remains to show the above assertion.

If $\{ \alpha_{22, k}\}_{k = 1}^{\infty}$ is unbounded. We choose a subsequence of $\{ \alpha_{22, k}\}_{k = 1}^{\infty}$, still denoted as $\{ \alpha_{22, k}\}_{k = 1}^{\infty}$, such that $\alpha_{22, k}\rightarrow {\infty}$ as $k\rightarrow {\infty}$. We can then argue in very much the same way as before to conclude that $v_{k}\rightarrow \hat{v}$ for some non-negative constant $\hat{v}$.

If $\{ \alpha_{22, k}\}_{k = 1}^{\infty}$ is bounded. Without loss of generality, we may assume that $\alpha_{22, k}\rightarrow \alpha_{22}\in [0, \infty)$. Set

Since $\{ \alpha_{22, k}\}_{k = 1}^{\infty}$ is bounded, by Lemma 3.4 it is easy to know that $\| \Upsilon_k \|_{L^\infty (\Omega)}\leq C$. Hence, $\Upsilon_k$ satisfies

By standard $L^p$ estimate and the Sobolev embedding theorem, we obtain $\| \Upsilon_k \|_{C^{1, \alpha} (\overline{\Omega})}\leq C$ for some $\alpha \in (0, 1)$. Then, by passing to a subsequence if necessary, we may assume that $\{\Upsilon_{k}\}_{k = 1}^{\infty}$ converges to some nonnegative function $\Upsilon$ in $C^{1} (\overline{\Omega})$. By the definition of $\Upsilon_{k}$ and the fact $u\rightarrow \hat{u}$, we see

in $C^{1} (\overline{\Omega })$. If $\alpha_{22} > 0$, it is easy to get $v_k\rightarrow \tilde{v}$ in $C^{1} (\overline{\Omega})$, where

Letting $k\rightarrow {\infty}$ in (4.4), we can know that $\Upsilon$ satisfies the following problem weakly

The standard subelliptic regularity theory ensures that $\Upsilon\in C^{2} (\overline{\Omega})$, and hence is a classical solution of (4.5). Note that $\Upsilon\geq0$. If $\Upsilon\equiv0$, then we claim that $v_k\rightarrow 0$ in $C^{1} (\overline{\Omega})$. Since $u_k\rightarrow \hat{u}$, by (4.2) we can argue similarly as before to show that $f(\hat{u}, 0) = 0$ and $\hat{u} > 0$, which contradicts (H2). Therefore, $\Upsilon\geq0$ and is not identically zero in $\Omega$. The problem (4.5) can be rewritten as

By Lemma 2.2, $\Upsilon > 0$, and thus $\tilde{v} > 0$ in $\overline{\Omega}$. Since $\tilde{v}$ is a solution of

by Lemma 2.4 and the positivity of $\tilde{v}$, we obtain $g(\hat{u}, \max\limits_{\overline{\Omega}}\tilde{v})\geq0$. Thus, from assumption (H1), it follows that

Integrating the equation of (4.6) in ${\Omega}$ shows

which implies that ${\tilde{v}}\equiv\max\limits_{\overline{\Omega}}\tilde{v} > 0$. That is, if $\alpha_{22, k}\rightarrow \alpha_{22} > 0$, then there exists a subsequence of $\{ \alpha_{22, k}\}_{k = 1}^{\infty}$ which converges uniformly to some positive constant.

If $\alpha_{22} = 0$, we have already established that

in $C^{1} (\overline{\Omega})$ as $k\rightarrow {\infty}$. Then, our conclusion that a subsequence of $\{ v_{k}\}_{k = 1}^{\infty}$ converges to some positive constant follows from the same arguments as in the case $\alpha_{22} > 0$ with obvious modifications. This proves our assertion, and the proof of part (ii) is now complete.

Finally, we return to the proof of part (ⅰ). Suppose that the conclusion in (ⅰ) fails. Then, we can assume that there exists a sequence of solutions $\{(u_k, v_k)\}_{k = 1}^{\infty}$ to (4.2) with $\alpha_{11, k}\rightarrow {\infty}$.

Similarly to the processes in part (ⅱ), we show that there exists a subsequence of $\{(u_k, v_k)\}_{k = 1}^{\infty}$ that converges uniformly to some non-negative $(\hat{u}, \hat{v})$. Again, (4.3) and the arguments following it guarantee that $f(\hat{u}, \hat{v}) = g(\hat{u}, \hat{v}) = 0$. By (H1) and (H2), we conclude that $\hat{u} > 0$ and $\hat{v} > 0$. However, this contradicts our assumption of (ⅰ). □

Lemma 4.3. Suppose that (H1) and (H2) hold and $\min\{d_1, d_2\}\geq \epsilon$.

(i) If $f(u, v) = g(u, v) = 0$ have no positive root, then there exists some positive constant $C_1 = C_1(\epsilon, \alpha_{11},$ $\alpha_{12}, \alpha_{21}, \alpha_{22})$ such that (1.2) has no solution provided that $\max\{d_1, d_2\}\geq C_1$.

(ii) If $f(u, v) = g(u, v) = 0$ have a positive root, then for any small $\varepsilon > 0$, there exists a positive constant $C_2 = C_2(\varepsilon, \epsilon, \alpha_{11}, \alpha_{12}, \alpha_{21}, \alpha_{22})$ such that if $\max\{d_1, d_2\}\geq C_2$, for any solution $(u, v)$ of (1.2), there are two positive constants $\hat{u}, \hat{v}$ that $f(\hat{u}, \hat{v}) = g(\hat{u}, \hat{v}) = 0$ and $\| u-\hat{u} \|_{L^\infty (\Omega)}+\| v-\hat{v} \|_{L^\infty (\Omega)}\leq \varepsilon$.

Proof. We shall only prove part (ⅱ), as (ⅰ) can be shown in a similar way. For the proof of (ⅱ), we still argue by contradiction. We assume that there exist two positive constants $\epsilon_0$ and $\varepsilon_0$, and a sequence $\{d_{1, k}, d_{2, k}\}_{k = 1}^{\infty}$ with $d_{1, k}\rightarrow {\infty}$ and $d_{2, k}\geq\epsilon_0$, such that

for any positive root $(\hat{u}, \hat{v})$ of $f(u, v) = g(u, v) = 0$, where $(u_k, v_k)$ is a solution to

For the problem (4.8), Lemma 3.4 implies that

To show that $u_k$ converges to some constant, let

Then by similar arguments as in the proof of Lemma 4.2, we see that $\Phi_k$ converges uniformly to some non-negative constant $\Phi$. By (4.9) and the fact $d_{1, k}\rightarrow {\infty}$, $u_k$ converges uniformly to $\Phi$. If $\{d_{2, k}\}_{k = 1}^{\infty}$ is unbounded, then it is easy to show that a subsequence of $\{d_{2, k}\}_{k = 1}^{\infty}$ also converges to a non-negative constant. If $\{d_{2, k}\}_{k = 1}^{\infty}$ bounded, setting

we know that a subsequence of $\{\Upsilon_{k}\}_{k = 1}^{\infty}$ converges to some non-negative function $\Upsilon$, and hence a subsequence of $\{v_{k}\}_{k = 1}^{\infty}$ converges to a nonnegative function $\tilde{v}$. Then, we can proceed further as in the proof of Lemma 4.2 to show that $\tilde{v}$ is a constant that derives a contradiction. □

Lemma 4.4. Suppose that (H1) and (H2) hold and $\alpha_{22} > 0$.

(i) If $f(u, v) = g(u, v) = 0$ has no positive root, then there exists a positive constant $C_3 = C_3(\alpha_{11}, \alpha_{12},$ $\alpha_{21}, \alpha_{22})$ such that (1.2) has no solution provided that $d_1\geq C_3$.

(ii) If $f(u, v) = g(u, v) = 0$ has a positive root, then for any small $\varepsilon > 0$, there exists a constant $C_4 = C_4(\varepsilon, \alpha_{11}, \alpha_{12}, \alpha_{21}, \alpha_{22})$ such that if $d_1\geq C_4$, for any solution $(u, v)$ of (1.2), there exist two positive constants $\hat{u}, \hat{v}$ that $f(\hat{u}, \hat{v}) = g(\hat{u}, \hat{v}) = 0$ and $\| u-\hat{u} \|_{L^\infty (\Omega)}+\| v-\hat{v} \|_{L^\infty (\Omega)}\leq \varepsilon$.

Similar results hold if $\alpha_{11} > 0$ and $d_2$ is large enough.

Proof. In view of Lemma 4.3, it suffices to consider the case $d_{1, k}\rightarrow {\infty}$ and $d_{2, k}\rightarrow 0$. For this case, by following the proof of Lemma 4.3, we obtain

and

for large $k$. Then, we can prove Lemma 4.4 in the same way as Lemma 4.3. □

Proof of Theorem 4.1. In view of part (ⅰ) of Lemma 4.2, we may assume that $f(u, v) = g(u, v) = 0$ has at least a positive root. Setting

By (H1) and (H2) we know

Choosing $\varepsilon = \frac{\delta}{2}$ in Lemma 4.2, there is a positive constant $C(\delta)$ and $C$ such that if $\max\{\alpha_{11}, \alpha_{22}\}\geq C(\delta)$, then for any solution $(u, v)$ of (1.2),

Without loss of generality, we may assume that $\alpha_{11}$ is sufficiently large. Let $(\bar{u}, \bar{v})$ be the average of $(u, v)$ in $\Omega$. Multiplying the first equation of problem (1.2) by $u-\bar{u}$ and integrating in $\Omega$, by the same arguments as in (3.20), we get

By Lemma 3.4, (4.10), and Poincaré's inequality, we have

Using (4.10) and Poincaré's inequality, we obtain

where $\lambda_1$ is the smallest positive eigenvalues of the sub-Laplace operator subject to the homogeneous Neumann boundary condition (see ^[1]). For the second equation of problem (2.1), we proceed as in (3.21) to obtain

By Lemma 4.2, for any small $\varepsilon$, there exists $C(\varepsilon)$ such that if $\alpha_{11}\geq C(\varepsilon)$, then

for some $(\hat{u}, \hat{v})\in \mathcal{S}$. And then

As $\bar{v}\geq \frac{\delta}{2}$, we know that for $\alpha_{11}\geq C(\varepsilon)$ and $\varepsilon$ small enough,

Therefore

Combining (4.12) and (4.14), we have

Choosing $\varepsilon$ small enough, for $\alpha_{11}$ sufficiently large, ${{\nabla }_{\mathbb{H}}}u = {{\nabla }_{\mathbb{H}}}v\equiv0$, then $(u, v)$ is constant. □

Theorem 4.5. Suppose that the conditions (H1) and (H2) hold. For any $\epsilon > 0$, there exists some positive constant $C_5 = C_5(\epsilon, \alpha_{11}, \alpha_{12}, \alpha_{21}, \alpha_{22})$ such that if $\min\{d_1, d_2\}\geq \epsilon$ and $\max\{d_1, d_2\}\geq C_5$, then problem (1.2) has no non-constant solution.

Proof. Replacing $\alpha_{11}\delta$ by $d_1$ in both (4.12) and (4.15), and following the proof of Theorem 4.1 with the help of Lemma 4.3 instead, we see immediately that this theorem holds. □

Theorem 4.6. Suppose that the conditions (H1) and (H2) hold.

(i) There exists a positive constant $C_6 = C_6(d_2, \alpha_{11}, \alpha_{12}, \alpha_{21}, \alpha_{22})$ such that if $d_1\geq C_6$, problem (1.2) has no non-constant solution. Furthermore, if $\alpha_{22} > 0$, then $C_6$ can be chosen independent of $d_2$.

(ii) There exists a positive constant $C_7 = C_7(d_1, \alpha_{11}, \alpha_{12}, \alpha_{21}, \alpha_{22})$ such that if $d_2\geq C_7$, problem (1.2) has no non-constant solution. Furthermore, if $\alpha_{11} > 0$, then $C_7$ can be chosen independent of $d_1$.

Proof. We shall establish part (ⅰ) only, since (ⅱ) can be shown in a similar way. By letting $\epsilon = d_2$ and $C_6 = \max\{C_5, d_2\}$ in Theorem 4.5, we know that the first assertion of (ⅰ) follows immediately from Theorem 4.5. To prove the second assertion, we first note that by choosing $\varepsilon = \frac{\delta}{2}$, from Lemma 4.4 it follows that $\min\limits_{\overline{\Omega}}v \geq \frac{\delta}{2}$. Then, we modify the proof of Theorem 4.1 by replacing the constant $d_2$ in (4.14) and (4.15) by $2\alpha_{22} \min\limits_{\overline{\Omega}}v$, and the term $\alpha_{11}\delta$ by $d_1$ in both (4.12) and (4.15). The remaining arguments are rather similar as before and are thus omitted. □

It follows immediately from Theorem 4.6 that

Corollary 4.7. Suppose that the conditions (H1) and (H2) hold, $\alpha_{11} > 0$ and $\alpha_{22} > 0$. Then, there exists a positive constant $C_8 = C_8(\alpha_{11}, \alpha_{12}, \alpha_{21}, \alpha_{22})$ such that if $\max\{d_1, d_2\}\geq C_8$, problem (1.2) has no non-constant solution.

Remark 2. We note that Theorem 1.1 follows from Theorem 4.1 and Corollary 4.7. Moreover, from Theorem 4.1 and Corollary 4.7 we see that large self-diffusion seems to have a very similar effect to large diffusion, as observed in ^[16].

5.   Predator-prey model

In this section, we mainly study the predator-prey system (1.3). Throughout this section, $C$ will always denote generic positive constants.

At first, we study the case $\bar{\alpha}_{21} = 0$, and give the proof of Theorem 1.2. As a by-product a priori estimate is established by using the maximum principle and the Harnack inequality.

Theorem 5.1. Suppose that $\tilde{d}_1, \tilde{d}_2,$ and $\tilde{\alpha}_{12}$ are given positive constants. Then, there exists a positive constant $C = C(a_2, c_2, \tilde{d}_1, \tilde{d}_2, \tilde{\alpha}_{12})$ such that if $d_1\geq \tilde{d}_1, d_2\geq \tilde{d}_2$ and $\bar{\alpha}_{12}\leq \tilde{\alpha}_{12}$, then every positive solution $(u, v)$ of (1.3) satisfies $C^{-1} < u, v < C$.

Proof. Assume that $(u, v)$ is a positive solution of problem (1.3) and denote $\Pi = (1+\alpha_{12}v)u$. Then problem (1.3) becomes

Let $\xi_1\in\overline{\Omega}$ be a point where $\Pi(\xi_1) = \max\limits_{\overline{\Omega}}\Pi(\xi)$. By Lemma 2.4, for the first equation of problem (5.1), we obtain that

Therefore, $u(\xi_1)q(u(\xi_1))\geq 0.$ By (H4), we have

and

here, the condition $\lim\limits_{u\rightarrow 0^+}p'(u) < \infty$ in (H5) shows that $\sup\limits_{u\in(0, S)}\frac{uq(u)}{p(u)} < \infty$. Thus,

Multiplying $c_2$ to the first equation of (1.3) and adding it to the second equation of (1.3) and then integrating over $\Omega$, we obtain

By Green's identity, we know that

So

that is,

The problem (1.3) can also be written as

For $u < S$ and $d_2\geq \tilde{d}_2$, we see $\frac{-a_2+c_2 p(u)}{d_2} < \frac{c_2 p(S)}{\tilde{d}_2} < \infty$, so the Lemma 2.5 holds for $v$,

for some positive constant $C_0$. Hence, we have

By integrating the first equation of problem (1.3) over $\Omega$, we have

Equation (5.4) implies that there exists a point $\xi_2\in\Omega$, such that

By assumptions (H4) and (H5), it follows that $0 < u(\xi_2) < S$. Then,

If $\min\limits_{\overline{\Omega}}v = 0$, by (5.2) it follows that $\max\limits_{\overline{\Omega}}v = 0$. That means that $v\equiv0$ uniformly in $\Omega$, which is a contradiction. Thus $v$ has a positive lower bound for $d_2\geq \tilde{d}_2$.

In the following, we show that $u$ has a positive lower bound.

By (H5) and $p(u)\in C^2((0, +\infty))$, it follows that

there exists a positive constant $\bar{p}$ such that $\frac{p(u)}{u}\leq \bar{p}$ for small $0 < u\leq S$. For $d_1\geq \tilde{d}_1$, we have

Thus Lemma 2.5 holds for $\Pi$,

for some positive constant $\tilde{C}_0$. By (5.3) and (5.5), we get

To obtain a contradiction, assume that there exists a sequence $\{(d_{1, k}, d_{2, k}, \bar{\alpha}_{12, k})\}_{k = 1}^{\infty}$, satisfying $d_{1, k}\geq \tilde{d}_1, d_{2, k}\geq \tilde{d}_2$ and $\bar{\alpha}_{12, k}\leq \tilde{\alpha}_{12}$ for some $\tilde{\alpha}_{12} > 0$, such that the corresponding positive solutions $(u_k, v_k)$ of problem (1.3) with $(d_1, d_2, \bar{\alpha}_{12}) = (d_{1, k}, d_{2, k}, \bar{\alpha}_{12, k})$ such that $\min\limits_{\overline{\Omega}} u_k\rightarrow0$ as $k\rightarrow \infty$. Using (5.6), we have $\max\limits_{\overline{\Omega}} u_k\rightarrow0$ as $k\rightarrow \infty$. By the regularity theory for subelliptic equations, there exists a subsequence of $\{(u_k, v_k)\}$, which will also be denoted by $\{(u_k, v_k)\}$, such that $u_k\rightarrow0$ uniformly as $k\rightarrow \infty$. Integrating the second equation of problem (1.3) with $(u, v) = (u_k, v_k)$, we obtain

Since $u_k\rightarrow0$ as $k\rightarrow \infty$, we have $-a_2+c_2 p(u_k) < 0$ in ${\overline{\Omega}}$ for any large $k$. This contradicts the integrating identity (5.7) as well as the fact that $v_k > 0$. □

Theorem 5.2. Suppose that $p(S)\leq \frac{a_2}{c_2}$, then problem (1.3) has no non-constant solution.

Proof. Since $q(u) < 0$ for $u\geq S$, we only need to consider the case $u < S$. Suppose, on the contrary, that (1.3) has a non-constant positive solution $(u, v)$ for $p(S)\leq \frac{a_2}{c_2}$. Then, $v$ must be non-constant; otherwise, it is easily seen that $u$ must be constant from the second equation of problem (1.3).

Using the fact that $p(u)$ is increasing in $u$, and integrating the second equation of problem (1.3) over $\Omega$, we have

Since $v > 0$, we have $p(S)\geq \frac{a_2}{c_2}$. This contradiction completes the proof. Thus, if (1.3) has a positive solution, it must be that $p(S)\geq \frac{a_2}{c_2}$, which is the condition that $p(u)$ should satisfy according to (H5). □

Remark 3. Theorem 5.2 is directly characterized by the function $p(u)$. When $\bar{\alpha}_{21} = 0$, we will prove the non-existence result. Theorem 1.2, which considers the self-diffusion and cross-diffusion rates $d_1,$ and $d_2$, is given in ^[14].

Theorem 5.3. Suppose that $\tilde{d}_1 = \lambda_1+\lambda_1^{-1}(q(0)+c_2\lambda_1 K)$ and $\tilde{d}_2 = \lambda_1^{-1}(-a_2+c_2p(\tilde{C})+\frac{c_2 K+(d_1\bar{\alpha}_{12} \tilde{C})^2}{4})$ where $K = \sup\limits_{\overline{\Omega}}\bar{v}p'(u)$. If $d_1\geq \tilde{d}_1$ and $d_2\geq \tilde{d}_2$, then problem (1.3) has no non-constant solution.

Proof. Let $(u, v)$ be a positive solution of problem (1.3). Multiplying the equations of problem (1.3) by $u-\bar{u}, v-\bar{v}$, and then integrating over $\Omega$, using the mean value theorem, we get

where $0 < \eta, \zeta, \varsigma\leq \tilde{C}$ and $K = \sup\limits_{\overline{\Omega}}\bar{v}p'(u)$, we note here that $p'(u)$ is bounded in any finite interval due to the assumptions $p(u)\in C^2((0, +\infty))$ and (H5).

By Theorem 5.1, Cauchy's inequality, and Poincaré's inequality, we see

where $T$ is taken as any positive constant, specifically $\lambda_1$. Hence, by the assumptions $d_1\geq \tilde{d}_1, d_2\geq \tilde{d}_2$, we know that problem (1.3) has no non-constant positive solution. □

The Theorem 1.2 can be obtained from Theorem 5.3.

Remark 4. If $\bar{\alpha}_{21} = 0$ and $\bar{\alpha}_{12}$ is small enough as ^[14], then Theorem 1.2 shows that problem (1.3) does not admit a non-constant positive solution for some large enough $d_1, \; d_2$, which is consistent with the result Theorem of 1.1.

Next, we prove Theorem 1.3.

Theorem 5.4. Suppose that $\tilde{d}_1, \tilde{d}_2, \tilde{\alpha}_{12}, \tilde{\alpha}_{21}$ are given positive constants. Then, there exists a positive constant $C = C(a_2, c_2, \tilde{d}_1, \tilde{d}_2, \tilde{\alpha}_{12}, \tilde{\alpha}_{21})$ such that if $d_1\geq \tilde{d}_1, d_2\geq \tilde{d}_2, \bar{\alpha}_{12}\leq \tilde{\alpha}_{12}$ and $\bar{\alpha}_{21}\leq \tilde{\alpha}_{21}$, then every positive solution $(u, v)$ of (1.3) satisfies $C^{-1} < u, v < C$.

The proof of Theorem 5.4 is similar to Theorem 5.1.

Theorem 5.5. Suppose that $\tilde{d}_1 = \lambda_1^{-1}q(0)+c_2 K$ and $\tilde{d}_2 = \lambda_1^{-1}(-a_2+\frac{c_2 K+\tilde{C}^2}{4\lambda_1})$ with $K = \sup\limits_{\overline{\Omega}}\bar{v}p'(u)$. Then, there exists positive constants $\tilde{d}_1, \tilde{d}_2, \tilde{\alpha}_{12}, \tilde{\alpha}_{21}$ such that if $d_1 > \tilde{d}_1$ and $d_2 > \tilde{d}_2$, then problem (1.3) has no non-constant solution when $\bar{\alpha}_{12} < \tilde{\alpha}_{12}$ and $\bar{\alpha}_{21} < \tilde{\alpha}_{21}$.

Proof. Let $(u, v)$ be a positive solution of problem (1.3). Multiplying the equations of problem (1.3) by $u-\bar{u}, v-\bar{v}$, and then integrating over $\Omega$, using the mean value theorem, we have

where $0 < \eta, \zeta, \varsigma\leq \tilde{C}$ and $K = \sup\limits_{\overline{\Omega}}\bar{v}p'(u)$; here, we note that $p'(u)$ is bounded in any finite interval in view of the assumptions $p(u)\in C^2((0, +\infty))$ and (H5).

By Theorem 5.1, the Cauchy's inequality, and Poincaré's inequality, we have

where $T$ is taken as any positive constant, specifically $\lambda_1$. Recall that $C_1 = (1+\bar{\alpha}_{12}M)S$ in the proof of Theorem 5.1. Hence,

i.e.,

we know that, under the given assumptions, Theorem 1.3 implies that problem (1.3) has no non-constant positive solution. □

The Theorem 1.3 can be obtained from Theorem 5.5.

Remark 5. If $\bar{\alpha}_{12}$ and $\bar{\alpha}_{21}$ are small enough as ^[28], then Theorem 1.3 shows that problem (1.3) does not admit a non-constant positive solution for some large enough $d_1, \; d_2$, which is consistent with the result of Theorem 1.1.

6.   Conclusions

We consider the Neumann boundary value problem for the strongly-coupled subelliptic system and the predator-prey subelliptic system on the Heisenberg group. We provide a priori estimates and non-existence results for non-constant positive solutions of the strongly-coupled and predator-prey systems with coefficients under different conditions. Only one of the diffusion rates or one of the self-diffusion pressures needs to be large to prevent the formation of non-constant solutions in the strongly-coupled subelliptic systems. For the predator-prey subelliptic system with cross-diffusion and homogeneous Neumann boundary conditions, we investigate the existence and non-existence of non-constant positive solutions.

Author contributions

Xinjing Wang: Investigation, Methodology, Validation, Writing-review and editing, Formal analysis; Guangwei Du: Methodology, Writing-original draft preparation, Visualization, Validation. All authors have read and agreed to the published version of the manuscript. Both authors contributed equally and significantly to this manuscript.

Acknowledgments

This work is supported by the Youth Backbone Teacher Funding Program in Huanghuai University, the Key Specialized Research and Development Breakthrough Program in Henan Province (No.222102310265), the Key Scientific Research Project of the Colleges and Universities in Henan Province (No.23B110009), and the China Scholarship Council (No.202408410284).

Conflict of interest

The authors declare no conflict of interest.