1 Introduction and main results

In this paper, we consider the following Schrödinger-Kirchhoff-Poisson system:

$$ \textstyle\begin{cases} -(a+b\int_{\Omega}|\nabla u|^{2}\,\mathrm{d}x)\triangle u+\lambda\phi u=\eta f(x,u)+u^{5}, &\mbox{in }\Omega, \\ -\triangle\phi=u^{2}, &\mbox{in }\Omega , \\ u=\phi=0, &\mbox{on } \partial\Omega, \end{cases} $$
(1.1)

where \(a\geq0\), \(b>0\) and \(\eta,\lambda>0\), \(\Omega\subset R^{3}\) is a bounded smooth domain.

When \(a=1\) and \(b=0\), the problem (1.1) reduces to the boundary value problem

$$ \textstyle\begin{cases} -\triangle u+\phi u=f(x,u), &\mbox{in }\Omega, \\ -\triangle\phi=u^{2}, &\mbox{in }\Omega , \\ u=\phi=0, &\mbox{on }\partial\Omega. \end{cases} $$
(1.2)

System (1.2) is relevant to the nonlinear parabolic Schrödinger-Poisson system:

$$ \textstyle\begin{cases} -i\frac{\partial\psi}{\partial t}=-\triangle\psi+\phi(x)\psi -|\psi|^{p-2}\psi, &\mbox{in }\Omega, \\ -\triangle\phi=|\psi|^{2}, &\mbox{in }\Omega , \\ \psi=\phi=0, &\mbox{on }\partial\Omega. \end{cases} $$
(1.3)

The first equation in (1.3) is called the Schrödinger equation, which describes (non-relativistic) quantum particles interacting with the electromagnetic field generated by the motion. An interesting class of Schrödinger equations is the case where the potential \(\phi(x)\) is determined by the charge of the wave function itself, that is, when the second equation in (1.3) (Poisson equation) holds. For more details as regards the physical relevance of the Schrödinger-Poisson system, we refer to [13].

System (1.2) has been extensively studied after the seminal work of Benci and Fortunato [3]. Many important results concerning existence and nonexistence of solutions, multiplicity of solutions, least energy solutions, and so on, have been reported; see for instance [412] and the references therein.

On the other hand, considering just the first equation in (1.1) with the potential equal to zero, we have the problem

$$ \textstyle\begin{cases} -(a+b\int_{\Omega}|\nabla u|^{2})\triangle u=f(x,u), &\mbox{in } \Omega, \\ u=0, &\mbox{on }\partial\Omega, \end{cases} $$
(1.4)

which was proposed by Kirchhoff in 1883 (see [13]) as a generalization of the well-known D’Alembert wave equation,

$$\rho\frac{\partial^{2}u}{\partial t^{2}}-\biggl(\frac{\rho_{0}}{h}+\frac {E}{2l} \int_{0}^{l}\biggl|\frac{\partial u}{\partial x}\biggr|^{2}\, \mathrm{d}x\biggr)\frac {\partial^{2}u}{\partial x^{2}}=g(x,u). $$

In recent years, with the aid of variational methods, the analysis of the stationary problem of (1.4) has been extensively carried out by many authors; see [1424] and so on. By them, several existence results have been successfully obtained via the variational and topological methods even for the critical case. But most of them consider only the one nonlocal term. In our case, we denote \(\varphi(u)=\int_{\Omega}\phi_{u}u^{2}\,\mathrm{d}x\), it follows from Lemma 2.1 in the following section that \(\varphi:H\rightarrow R\) is \(C^{1}\) and \(\varphi(tu)=t^{4}\varphi(u)\). Although φ is a 4 homogeneous function, it is not equivalent to the nonlinear function \(f(u)=u^{4}\). Therefore our problem (1.1) possesses two nonlocal terms, \(\phi_{u}u\) and \(b(\int_{\Omega}|\nabla u|)\triangle u\). A typical difficulty occurs in proving the existence of solutions. It is caused by the lack of the compactness of the Sobolev embedding \(H_{0}^{1}(\Omega)\hookrightarrow L^{6}(\Omega)\). Furthermore, in view of the corresponding energy, the interaction between the Kirchhoff type perturbation \(\|u\|_{H_{0}^{1}(\Omega)}^{4}\) and the critical nonlinearity \(\int_{\Omega}u^{6}\,\mathrm{d}x\) is crucial. In the following, we can see the effect of such an interaction on the existence. To the best of our knowledge, there is little literature which essentially attacks the Brezis-Nirenberg problem for Schrödinger-Kirchhoff-Poisson type with critical nonlinearity equations.

Motivated by the above facts, the goal of this paper is to consider the existence of non-trivial solutions for problem (1.1). Under some natural assumptions, by using the mountain pass theory, the existence results of non-trivial solutions are obtained.

Before stating our main results, we give the following assumption.

The hypotheses on the function \(f:\Omega\times\mathbb{R}\rightarrow \mathbb{R}\) are the following.

(F1):

\(\lim_{t\rightarrow0}\frac{f(x,t)}{t}=0\), uniformly on \(x\in\Omega\).

(F2):

\(\lim_{t\rightarrow0}\frac{f(x,t)}{t^{5}}=0\), uniformly on \(x\in\Omega\).

(F3):

The well-known Ambrosetti-Rabinowitz superlinear condition, that is, \(0<\theta F(x,t)=\theta\int_{0}^{t}f(x,s)\,\mathrm{d}s\leq tf(x,t)\) for all \(x\in\Omega\) and \(t>0\) for some \(4<\theta<6\) and \(F(x,t)=\int _{0}^{t}f(x,s)\,\mathrm{d}s\).

(F4):

\(\lim_{t\rightarrow0}\frac{f(x,t)}{t^{3}}=0\), uniformly on \(x\in\Omega\).

Now we state our main results.

Theorem 1.1

For all \(\eta\geq\overline{\eta}\) and \(\lambda\in(0,\lambda^{\ast})\), where \(\eta, \lambda ^{\ast}\in R\). Suppose that (F1), (F2), (F3) are satisfied, then problem (1.1) has at least a positive non-trivial solution.

Corollary 1.2

Suppose that \(a=0\) and (F1), (F2), (F4) are satisfied, then the problem (1.1) has at least a positive non-trivial solution.

Remark 1.3

For (1.1), when \(\lambda=0\), the problem (1.1) reduces to the Kirchhoff equation, when \(b=0\), the problem (1.1) is the Schrödinger-Poisson system equation.

Remark 1.4

If \(a=0\), (1.1) is a degenerate case, we claim (1.1) is the degenerate Schrödinger-Kirchhoff-Poisson system.

The remainder of this paper is organized as follows. In Section 2, some preliminary results are presented. In Section 3, we give the proof of our main results.

2 Variational setting and preliminaries

In this section, we collect some information to be used in the paper. Hereafter we use the following notations:

  • \(H^{1}(\mathbb{R}^{3})\) is the usual Sobolev space endowed with the standard scalar product and norm

    $$ (u,v)= \int_{\mathbb{R}^{3}}\nabla u\nabla v\,\mathrm{d}x,\qquad \|u\| ^{2}= \int_{\mathbb{R}^{3}}|\nabla u|^{2}\,\mathrm{d}x. $$
    (2.1)
  • S denotes the best Sobolev constant \(S:=\inf_{u\in D^{1,2}(\mathbb{R}^{3})\setminus\{0\}}\frac{\int_{\mathbb {R}^{3}|\nabla u|^{2}\,\mathrm{d}x}}{(\int_{\mathbb {R}^{3}}u^{6}\,\mathrm{d}x)^{\frac{1}{3}}}\).

  • \(H^{\ast}\) denotes the dual space of \(H^{1}(\mathbb {R}^{3})\).

Now we define a functional I on H by

$$\begin{aligned} I(u) =&\frac{a}{2} \int_{\Omega}|\nabla u|^{2}\,\mathrm{d}x+ \frac {b}{4}\biggl( \int_{\Omega}|\nabla u|\,\mathrm{d}x\biggr)^{2}+ \frac{\lambda }{4} \int_{\Omega}\phi u^{2}\,\mathrm{d}x \\ &{}-\frac{1}{6} \int_{\Omega }u^{6}\,\mathrm{d}x-\eta \int_{\Omega}F(x,u)\,\mathrm{d}x. \end{aligned}$$
(2.2)

It is easy to prove that the functional I is of class \(C^{1}(H,\mathbb{R})\). Moreover,

$$\begin{aligned} \bigl\langle I'(u),v\bigr\rangle =&a \int_{\Omega}|\nabla u|\nabla v\,\mathrm {d}x+b\|u\|^{2} \int_{\Omega}|\nabla u||\nabla v|\,\mathrm{d}x- \int _{\Omega}u^{5}v\,\mathrm{d}x \\ &{}+\lambda \int_{\Omega}\phi uv\,\mathrm {d}x-\eta \int_{\Omega}f(x,u)v\,\mathrm{d}x. \end{aligned}$$
(2.3)

The following result is well known (see e.g. [1, 5, 15]).

Lemma 2.1

For each \(u\in H_{0}^{1}(\Omega)\), there exists a unique element \(\phi_{u}\in H_{0}^{1}(\Omega)\) such that \(-\triangle\phi_{u}=u^{2}\), moreover, \(\phi_{u}\) has the following properties:

  1. (a)

    there exists \(c>0\) such that \(\|\phi_{u}\|\leq c\|u\| ^{2}\) and

    $$ \int_{\Omega}|\nabla\phi_{u}|^{2}\, \mathrm{d}x= \int_{\Omega}\phi _{u}u^{2}\,\mathrm{d}x\leq c \|u\|^{4}; $$
    (2.4)
  2. (b)

    \(\phi_{u}\geq0\) and \(\phi_{tu}=t^{2}\phi_{u}\), \(\forall t>0\);

  3. (c)

    if \(u_{n}\rightharpoonup u\) in \(H_{0}^{1}\), then \(\phi_{u_{n}}\rightharpoonup\phi_{u}\) in \(H_{0}^{1}\) and

    $$ \lim_{n\rightarrow+\infty} \int_{\Omega}\phi _{u_{n}}u^{2}_{n}\, \mathrm{d}x= \int_{\Omega}\phi_{u}u^{2}\,\mathrm {d}x. $$
    (2.5)

3 Proof of Theorem 1.1

We show that the functional I has the mountain pass geometry.

Lemma 3.1

Suppose that (F1), (F2) and (F3) hold, then we have:

  1. (i)

    There exist \(r,\rho>0\), such that \(\inf_{\|u\| =r}I(u)\geq\rho>0\).

  2. (ii)

    There exists a nonnegative function \(e\in H_{0}^{1}(\Omega )\) such that \(\|e\|>r\) and \(I(e)<0\).

Proof

By (F1) and (F2), we have

$$F(x,t)\leq\frac{\varepsilon}{2}|u|^{2}+\frac{1}{q}c_{\varepsilon}|u|^{6}. $$

According to (2.2)

$$I(u)=\frac{a}{2}\|u\|^{2}+\frac{b}{4}\|u \|^{4}+\frac{\lambda }{4} \int_{\Omega}\phi u^{2}\,\mathrm{d}x-\frac{1}{6} \int_{\Omega }u^{6}\,\mathrm{d}x-\eta \int_{\Omega}F(x,u)\,\mathrm{d}x. $$

By the Sobolev theorem, there exists \(c_{1}>0\) such that

$$\begin{aligned} I(u)&\geq\frac{a}{2}\|u\|^{2}+\frac{b}{4}\|u \|^{4}+\frac{\lambda }{4} \int_{\Omega}\phi u^{2}\,\mathrm{d}x- \frac{1}{6}c_{1}\|u\| ^{6}-\eta\frac{\varepsilon}{2} \int_{\Omega}|u|^{2}\,\mathrm {d}x-\frac{\eta c_{\varepsilon}}{6} \int_{\Omega}|u|^{6}\,\mathrm{d}x \\ &\geq c_{2}\|u\|^{2}-\frac{1}{6}c_{1}\|u \|^{6}. \end{aligned}$$
(3.1)

So there exists \(\rho>0\) such that

$$\rho:=\inf_{\|u\|=r}\varphi(u)>0=\varphi(0),\quad \mbox{it satisfies (i)}. $$

It follows from (2.4) that

$$\begin{aligned} I(tu)&=\frac{at^{2}}{2}\|u\|^{2}+\frac{bt^{4}}{4}\|u \|^{4}+\frac {\lambda t^{4}}{4} \int_{\Omega}\phi u^{2}\,\mathrm{d}x-\frac {t^{6}}{6} \int_{\Omega}u^{6}\,\mathrm{d}x-\eta \int_{\Omega }F(x,tu)\,\mathrm{d}x \\ &\leq\frac{at^{2}}{2}\|u\|^{2}+\frac{bt^{4}}{4}\|u \|^{4}+\frac {\lambda t^{4}}{4}c\|u\|^{4}-\frac{t^{6}}{6} \int_{\Omega }u^{6}\,\mathrm{d}x. \end{aligned}$$

Hence for t big enough, there exists \(t_{0}u_{0}\) such that \(I(t_{0}u_{0})<0\), we take \(t_{0}u_{0}=e\) and \(I(e)<0\), we complete the proof. □

Recall that S is attained by the functions \(\frac{\varepsilon ^{\frac{1}{4}}}{\varepsilon+|x|^{2}}\), where \(\varepsilon>0\). Define \(V_{\varepsilon}(x)=\frac{\psi(x)\varepsilon^{\frac {1}{4}}}{\varepsilon+|x|^{2}}\), where \(\psi\in C_{0}^{\infty }(B_{2r}(0))\) such that \(0\leq\psi(x)\leq1\) and \(\psi(x)=1\) on \(B_{r}(0)\). From [25] we know that for \(\varepsilon>0\) small

$$\begin{aligned}& S+C_{1}\varepsilon^{\frac{1}{2}} \leq \int_{R^{3}}|\nabla V_{\varepsilon}|^{2}\, \mathrm{d}x=S+C_{2}\varepsilon^{\frac{1}{2}},\qquad \int_{\mathbb{R}^{3}}|V_{\varepsilon}|^{6}\,\mathrm{d}x=1, \end{aligned}$$
(Q1)
$$\begin{aligned}& C_{3}\varepsilon^{\frac{t}{4}} \leq \int_{\Omega}|V_{\varepsilon }|^{t}\,\mathrm{d}x\leq C_{4}\varepsilon^{\frac{t}{4}}, \quad 1\leq t< 3, \end{aligned}$$
(Q2)
$$\begin{aligned}& C_{3}\varepsilon^{\frac{t}{4}}|\ln\varepsilon| \leq \int_{\Omega }|V_{\varepsilon}|^{t}\,\mathrm{d}x\leq C_{4}\varepsilon^{\frac {t}{4}}|\ln\varepsilon|,\quad t=3, \end{aligned}$$
(Q3)
$$\begin{aligned}& C_{3}\varepsilon^{\frac{6-t}{4}} \leq \int_{\Omega}|V_{\varepsilon }|^{t}\,\mathrm{d}x\leq C_{4}\varepsilon^{\frac{6-t}{4}},\quad 3\leq t< 6. \end{aligned}$$
(Q4)

Lemma 3.2

Suppose that (F1), (F2), (F3) are satisfied; then for the problem (1.1), there exists u such that \(\sup_{t\geq0}I(tu)\leq\Gamma\), where \(\Gamma=[\frac{(b+\lambda c)^{2}}{12}s^{4}+\frac{a}{3}s]\frac {(b+\lambda c)s^{2}+\sqrt{(b+\lambda c)s^{4}+4as}}{2}+\frac {ab+\lambda ac}{12}s^{3}\).

Proof

Let \(u_{\varepsilon}\in C_{0}^{\infty}(R^{3})\) with \(u>0\) on Ω. We have, for \(t\geq0\),

$$\begin{aligned} I(tu_{\varepsilon}) =&\frac{at^{2}}{2} \int_{\Omega}|\nabla u_{\varepsilon}|^{2}\,\mathrm{d}x+ \frac{bt^{4}}{4}\biggl( \int_{\Omega }|\nabla u_{\varepsilon}|\,\mathrm{d}x \biggr)^{2}+\frac{\lambda t^{4}}{4} \int_{\Omega}\phi_{u_{\varepsilon}}u_{\varepsilon }^{2}\, \mathrm{d}x \\ &{}-\frac{t^{6}}{6} \int_{\Omega}u_{\varepsilon }^{6}\,\mathrm{d}x- \eta \int_{\Omega}F(x,tu_{\varepsilon})\,\mathrm{d}x \\ \leq&\frac{at^{2}}{2}\|u_{\varepsilon}\|^{2}\,\mathrm{d}x+\biggl( \frac {b+\lambda c}{4}\biggr)t^{4}\|u_{\varepsilon}\|^{4}- \frac{t^{6}}{6}-\eta \int_{\Omega}F(x,tu_{\varepsilon})\,\mathrm{d}x. \end{aligned}$$

We set \(h(t)=\frac{at^{2}}{2}\|u_{\varepsilon}\|^{2}+(\frac {b+\lambda c}{4})t^{4}\|u_{\varepsilon}\|^{4}-\frac{t^{6}}{6}\) and by \(h'(t)=0\) we have

$$\begin{aligned}& at \|u_{\varepsilon}\|^{2}+(b+\lambda c)t^{3} \|u_{\varepsilon}\| ^{4}-t^{5}=0, \end{aligned}$$
(3.2)
$$\begin{aligned}& t^{2} =\frac{(b+\lambda c)\|u_{\varepsilon}\|^{4}+\sqrt{(b+\lambda c)^{2}\|u_{\varepsilon}\|^{8}+4a\|u_{\varepsilon}\|^{2}}}{2}. \end{aligned}$$
(3.3)

Then it follows from (3.2) that

$$ a\|u_{\varepsilon}\|^{2}+(b+\lambda c)t^{2} \|u_{\varepsilon}\| ^{4}=t^{4}. $$
(3.4)

Combining (3.3) and (3.4), we can obtain

$$\begin{aligned} I(tu_{\varepsilon}) \leq&t^{2}\biggl[-\frac{a\|u_{\varepsilon}\| ^{2}+(b+\lambda c)t^{2}\|u_{\varepsilon}\|^{4}}{6}+ \frac{b+\lambda c}{4}t^{2}\|u_{\varepsilon}\|^{4}+ \frac{a}{2}\|u_{\varepsilon}\| ^{2}\biggr]-\eta \int_{\Omega}F(x,tu_{\varepsilon})\,\mathrm{d}x \\ =&\frac{b+\lambda c}{12}t^{4}\|u_{\varepsilon}\|^{4}+ \frac {a}{3}t^{2}\|u_{\varepsilon}\|^{2}-\eta \int_{\Omega }F(x,tu_{\varepsilon})\,\mathrm{d}x \\ =&\frac{(b+\lambda c)^{2}}{12}t^{2}\|u_{\varepsilon}\|^{8}+a \frac {(b+\lambda c)}{12}\|u_{\varepsilon}\|^{6}+\frac{a}{3}t^{2} \| u_{\varepsilon}\|^{2}-\eta \int_{\Omega}F(x,tu_{\varepsilon })\,\mathrm{d}x \\ =&\biggl[\frac{(b+\lambda c)^{2}}{12}\|u_{\varepsilon}\|^{8}+ \frac{a}{3}\| u_{\varepsilon}\|^{2}\biggr]\frac{(b+\lambda c)\|u_{\varepsilon}\| ^{4}+\sqrt{(b+\lambda c)\|u_{\varepsilon}\|^{8}+4a\|u_{\varepsilon}\| ^{2}}}{2} \\ &{}+\frac{(ab+a\lambda c)}{12}\|u_{\varepsilon}\|^{6}-\eta \int _{\Omega}F(x,tu_{\varepsilon})\,\mathrm{d}x. \end{aligned}$$
(3.5)

It is easy to verify the following inequality:

$$(\alpha+\beta)^{\theta}\leq\alpha^{\theta}+\theta(\alpha +1)^{\theta-1}\beta,\quad \alpha>0,0\leq\beta\leq1,\theta\geq1. $$

So we have

$$\begin{aligned}& \|u_{\varepsilon}\|^{4}\leq S^{2}+C\varepsilon^{\frac{1}{2}}, \\& \|u_{\varepsilon}\|^{6}\leq S^{3}+C\varepsilon^{\frac{1}{2}}, \\& \|u_{\varepsilon}\|^{8}\leq S^{4}+C\varepsilon^{\frac{1}{2}}, \end{aligned}$$

by (F1) and (F2), we obtain

$$\bigl\vert f(x,t)\bigr\vert \leq\varepsilon t^{5}+d(\varepsilon)t, \quad d(\varepsilon)>0. $$

So we obtain

$$ a\|u_{\varepsilon}\|^{2}+b\|u_{\varepsilon}\|^{4}t^{2} \leq t_{\varepsilon}^{4}+\varepsilon \int_{\Omega}|t_{\varepsilon }|^{4}|u_{\varepsilon}|^{6} \,\mathrm{d}x+d(\varepsilon) \int_{\Omega }|u_{\varepsilon}|^{2}\, \mathrm{d}x $$
(3.6)

and

$$ t^{4}_{\varepsilon}+\varepsilon \int_{\Omega}|t_{\varepsilon }|^{4}|u_{\varepsilon}|^{6} \,\mathrm{d}x =t_{\varepsilon}^{4}\biggl(1+\varepsilon \int_{\Omega}|u_{\varepsilon }|^{6}\,\mathrm{d}x\biggr) \leq\frac{3}{2}t_{\varepsilon}^{4}. $$
(3.7)

For ε small enough, it follows that \(d(\varepsilon)\int_{\Omega}|u_{\varepsilon}|^{2}\,\mathrm {d}x\rightarrow0\) as \(\varepsilon\rightarrow0\) and that \(d(\varepsilon)\int_{\Omega}|u_{\varepsilon}|^{2}\,\mathrm{d}x\leq a\|u_{\varepsilon}\|^{2}\), combining (3.6) and (3.7) we obtain

$$a\|u_{\varepsilon}\|^{2}+b\|u_{\varepsilon}\|^{4}\leq \frac {3}{2}t_{\varepsilon}^{4}+a\|u_{\varepsilon} \|^{2}. $$

It follows that \(bs^{2}\leq b\|u_{\varepsilon}\|^{4}\leq\frac {3}{2}t_{\varepsilon}^{2}\).

This implies that \(t^{2}_{\varepsilon}\geq\frac{2}{3}bs^{2}\). so we can get \(F(x,t)\geq c_{0}\|t\|^{q}\).

It follows from (3.5) that

$$\begin{aligned} I(tu_{\varepsilon}) \leq&\biggl[\frac{(b+\lambda c)^{2}}{12}\bigl(S^{4}+C \varepsilon^{\frac{1}{2}}\bigr)+\frac{a}{3}\| u_{\varepsilon} \|^{2}\biggr] \\ &{}\times\frac{(b+\lambda c)(S^{2}+C\varepsilon^{\frac {1}{2}})+\sqrt{(b+\lambda c)(S^{4}+C\varepsilon^{\frac {1}{2}})+4a(S^{2}+C\varepsilon^{\frac{1}{2}})}}{2} \\ &{}+\frac{(ab+a\lambda c)}{12}\bigl(S^{3}+C\varepsilon^{\frac {1}{2}} \bigr)-c_{0}\biggl(\frac{2b}{3}s^{2} \biggr)^{\frac{q}{2}} \int_{\Omega }|u_{\varepsilon}|^{q}\,\mathrm{d}x, \end{aligned}$$

since \(\int_{\Omega}|u_{\varepsilon}|^{q}\,\mathrm{d}x\geq c_{2}\varepsilon^{\frac{6-q}{4}}\), by (F2), \(q>5\), we obtain \(\frac{1}{4}>\frac{6-q}{4}\). As ε is small enough, we obtain

$$\sup_{\varepsilon\rightarrow0}I(tu_{\varepsilon})< \biggl[\frac {(b+\lambda c)^{2}}{12}s^{4}+ \frac{a}{3}s\biggr]\frac{(b+\lambda c)s^{2}+\sqrt{(b+\lambda c)s^{4}+4as}}{2}+\frac{ab+\lambda ac}{12}s^{3}, $$

we take \(tu_{\varepsilon}=tu\), we can get \(\sup_{t\geq 0}I(tu)\leq\Gamma\), the proof is completed. □

Lemma 3.3

If the conditions (F1), (F2), and (F3) hold then there exists \(\overline{\eta}>0\) such that \(c_{\ast}\) belongs to the interval \((0,\Gamma)\) for all \(\eta\geq\overline{\eta}\).

Proof

If \(u_{0}\) is the function given by Lemma 3.2, it follows that there exists \(t_{\eta}>0\) verifying \(I(t_{\eta }u_{0})=\max_{t\geq0}I(tu_{0})\). Hence

$$at_{\eta}^{2}\|u_{0}\|^{2}+b t_{\eta}^{4}\|u_{0}\|^{4}+\lambda t^{4}_{\eta} \int_{\Omega}\phi u^{2}\,\mathrm{d}x=\eta \int_{\Omega }f(x,t_{\eta}u_{0})t_{\eta}u_{0} \,\mathrm{d}x+t_{\eta}^{6} \int _{\Omega}u_{0}^{6}\,\mathrm{d}x. $$

From (2.3)

$$at_{\eta}^{2}\|u_{0}\|^{2}+b t_{\eta}^{4}\|u_{0}\|^{4}+\lambda t^{2}_{\eta} \int_{\Omega}\phi u^{2}\,\mathrm{d}x\geq t_{\eta }^{6} \int_{\Omega}u_{0}^{6}\,\mathrm{d}x, $$

which implies that \(t_{\eta}\) is bounded. Thus, there exist a sequence \(\eta_{n}\rightarrow+\infty\) and \(t_{0}\geq 0\) such that \(t_{\eta_{n}}\rightarrow t_{0}\), as \(n\rightarrow+\infty \). Consequently, there is \(M>0\) such that

$$at_{\eta}^{2}\|u_{0}\|^{2}+b t_{\eta}^{4}\|u_{0}\|^{4}+\lambda t^{2}_{\eta} \int_{\Omega}\phi u^{2}\,\mathrm{d}x\leq M,\quad \forall n\in N, $$

and so

$$\eta \int_{\Omega}f(x,t_{\eta}u_{0})t_{\eta}u_{0} \,\mathrm {d}x+t_{\eta}^{6} \int_{\Omega}u_{0}^{6}\,\mathrm{d}x\leq M, \quad \forall n\in N. $$

If \(t_{0}>0\) the last inequality leads to

$$\lim_{n\rightarrow+\infty}\eta_{n} \int_{\Omega}f(x,t\eta _{n}u_{0})t \eta_{n}u_{0}\,\mathrm{d}x+t_{\eta_{n}}^{6} \int_{\Omega }u_{0}^{6}\,\mathrm{d}x=+\infty, $$

which is absurd.

Thus we conclude that \(t_{0}=0\), now consider the path \(\gamma(t)=te\), for \(t\in[0,1]\), which belongs to Γ and we get the following estimate:

$$\begin{aligned} 0 < &c_{\ast}\leq\max_{t\in[0,1]}I\bigl(\gamma(t)\bigr) =I(t_{\lambda}u_{0}) \\ \leq&\frac{a}{2}t_{\eta}^{2} \|u_{0}\|^{2}+\frac {b}{4} t_{\eta}^{4} \|u_{0}\|^{4}+\frac{\lambda}{4} t^{4}_{\eta} \int _{\Omega}\phi u^{2}\,\mathrm{d}x \\ &{}-t_{\eta}^{6} \int_{\Omega}u_{0}^{6}\,\mathrm{d}x-\eta \int_{\Omega }f(x,t_{\eta}u_{0})t_{\eta}u_{0} \,\mathrm{d}x. \end{aligned}$$

In this way, if η is large enough we derive \(I(t_{\eta }u_{0})<\Gamma\), which leads to \(0< c_{\ast}<\Gamma\). □

Proof of Theorem 1.1

We first prove that \(\{u_{n}\}\) is bounded in E:

$$\begin{aligned} c_{\ast}+1+\|u_{n}\| \geq&I(u_{n})- \frac{1}{\theta}\bigl\langle I'(u_{n}),u_{n} \bigr\rangle \\ =&a\biggl(\frac{1}{2}-\frac{1}{\theta}\biggr)\|u_{n} \|^{2}+b\biggl(\frac{1}{4}-\frac {1}{\theta}\biggr) \|u_{n}\|^{4} \\ &{}+\lambda\biggl(\frac{1}{4}- \frac{1}{\theta}\biggr) \int_{\Omega}\phi _{u_{n}}u_{n}^{2}\, \mathrm{d}x \\ &{}+\biggl(\frac{1}{\theta}-\frac{1}{6}\biggr) \int_{\Omega}|u_{n}|^{6}\,\mathrm{d}x + \frac{\eta}{\theta} \int_{\Omega}\bigl(f(x,u_{n})u_{n}-F(x,u_{n}) \theta \bigr)\,\mathrm{d}x \\ \geq&a\biggl(\frac{1}{2}-\frac{1}{\theta}\biggr)\|u_{n} \|^{2}, \end{aligned}$$

which implies that \(\{u_{n}\}\) is bounded.

Finally, we show that \(\{u_{n}\}\) possesses a strong convergent subsequence. Hence we assume that \(\{u_{n}\}\) is a \((\mathit{PS})_{c}\) sequence, for \(c\in (0,\Gamma)\),

$$ I(u_{n})\rightarrow c_{\ast} ,\qquad I'(u_{n}) \rightarrow0 , \quad n\rightarrow \infty. $$
(3.8)

Since \(\{u_{n}\}\) is bounded in \(H_{0}^{1}(\Omega)\) going if necessary to a subsequence, we assume that

$$\begin{aligned}& u_{n} \rightharpoonup u \quad \mbox{in } H_{0}^{1}( \Omega), \end{aligned}$$
(3.9)
$$\begin{aligned}& u_{n} \rightarrow u \quad \mbox{in } L^{(p)}(\Omega)\ (2\leq p< 6), \end{aligned}$$
(3.10)
$$\begin{aligned}& u_{n} \rightarrow u \quad \mbox{a.e in } \Omega, \end{aligned}$$
(3.11)

we write \(v_{n}=u_{n}-u\). It follows that

$$\begin{aligned}& \|u_{n}\|^{2} =\|v_{n}\|^{2}+\|u \|^{2}+o(1), \end{aligned}$$
(3.12)
$$\begin{aligned}& \|u_{n}\|^{4} =\|v_{n}\|^{4}+\|u \|^{4}+2\|v_{n}\|^{2}\|u\|^{2}+o(1), \end{aligned}$$
(3.13)

by (2.5), we have

$$ \int_{\Omega}\phi_{u_{n}}u^{2}_{n}\, \mathrm{d}x= \int_{\Omega}\phi _{u}|u|^{2}\, \mathrm{d}x+o(1). $$
(3.14)

The Brezis-Lieb lemma in [25] leads to

$$\int_{\Omega}u^{6}\,\mathrm{d}x= \int_{\Omega}v_{n}^{6}\,\mathrm {d}x+ \int_{\Omega}|u|^{6}\,\mathrm{d}x+o(1). $$

Making use of the Vitali convergence theorem, we obtain

$$\lim_{n\rightarrow\infty} \int_{R^{3}}f(x,u_{n})u_{n}\,\mathrm {d}x= \int_{R^{3}}f(x,u)u\,\mathrm{d}x. $$

If \(v_{n}=u_{n}-u\) and \(\|v_{n}\|\rightarrow0\), the proof is completed. Otherwise there exists a subsequence (still denoted by \(v_{n}\)) such that \(\lim_{n\rightarrow\infty}\|v_{n}\|=k\), where k is a positive constant.

Thus by \(I'(u_{n})\rightarrow0\) in \((H^{1}_{0})^{\ast}\), it follows that

$$\begin{aligned}& a\|v_{n}\|^{2}+a\|u\|^{2}+b\|v_{n} \|^{4}+b\|u\|^{4} \\& \quad {}+2b \int_{\Omega }|\nabla v_{n}|^{2}\,\mathrm{d}x \int_{\Omega}|\nabla u|^{2}\,\mathrm {d}x+\lambda \int_{\Omega}\phi_{u}u^{2}\,\mathrm{d}x \\& \quad {}- \int_{\Omega }|u|^{6}\,\mathrm{d}x- \int_{\Omega}|v_{n}|^{6}\,\mathrm{d}x-\eta \int_{\Omega }f(x,u)u\,\mathrm{d}x=o(1). \end{aligned}$$
(3.15)

It also follows from (3.8) that

$$\begin{aligned} \lim_{n\rightarrow\infty}\bigl\langle I'(u_{n}),u \bigr\rangle =&\bigl(a+b\|u_{n}\|^{2}\bigr) \int_{\Omega}\nabla u_{n}\nabla u\,\mathrm {d}x+\lambda \int_{\Omega}\phi_{u}u_{n}u\,\mathrm{d}x \\ &{}- \int_{\Omega }|u_{n}|^{5}u\,\mathrm{d}x-\eta \int_{\Omega}f(x,u_{n})u_{n}\,\mathrm {d}x \\ =&a\|u\|^{2}+bk^{2}\|u\|^{2}+b\|u\|^{4}+ \lambda \int_{\Omega}\phi _{u}u^{2}\,\mathrm{d}x \\ &{}- \int_{\Omega}|u|^{6}\,\mathrm{d}x-\eta \int _{\Omega}f(x,u)u\,\mathrm{d}x. \end{aligned}$$
(3.16)

On the one hand, by (3.16), we obtain

$$\begin{aligned} I(u)&=\frac{a}{2}\|u\|^{2}+\frac{b}{4}\|u \|^{4}+\frac{\lambda }{4} \int_{\Omega}\phi_{u}u^{2}\,\mathrm{d}x - \frac{1}{6} \int_{\Omega}u^{6}\,\mathrm{d}x-\eta \int_{\Omega }f(x,u)u\,\mathrm{d}x \\ &=\frac{1}{12} \int_{\Omega}u^{6}\,\mathrm{d}x+\eta \int_{\Omega }\biggl(\frac{1}{4}f(x,u)u-F(x,u)\biggr)\, \mathrm{d}x+\frac{a}{4}\|u\|^{2}-\frac {b}{4}k^{2} \|u\|^{2} \\ &>\frac{a}{4}\|u\|^{2}-\frac{b}{4}k^{2}\|u \|^{2}. \end{aligned}$$
(3.17)

On the other hand, it follows from (3.15) and (3.16) that

$$ I(u_{n})=I(u)+\frac{a}{2}\|v_{n}\|^{2}+ \frac{b}{4}\|v_{n}\|^{4}+\frac {b}{2} \|v_{n}\|^{2}\|u\|^{2}-\frac{1}{6} \int_{\Omega }|v_{n}|^{6}\,\mathrm{d}x+o(1) $$
(3.18)

and

$$ a\|v_{n}\|^{2}+b\|v_{n}\|^{4}+b \|v_{n}\|^{2}\|u\|^{2}- \int_{\Omega }|v_{n}|^{6}\,\mathrm{d}x=o(1). $$
(3.19)

By (3.19) and \(\int_{\Omega}|v_{n}|^{6}\,\mathrm{d}x\leq\frac{\| v_{n}\|^{6}}{s^{3}}\), we obtain

$$ak^{2}+bk^{2}\|u\|^{2}+bk^{4}\leq \frac{k^{6}}{s^{3}}, $$

so we have

$$ k^{2}\geq\frac{bs^{3}+\sqrt{b^{2}s^{6}+4(a+b\|u\|^{2})s^{3}}}{2}. $$
(3.20)

It follows from (3.19) and (3.20) that

$$I(u)=I(u_{n})-\frac{1}{3}a\|v_{n}\|^{2}- \frac{1}{12}b\|v_{n}\| ^{4}-\frac{1}{3}b \|v_{n}\|^{2}\|u\|^{2}, $$

let \(n\rightarrow\infty\), which implies that

$$\begin{aligned} I(u) =&c_{\ast}-\frac{1}{12}ak^{2}-\frac{1}{12}bk^{2}- \frac {1}{3}bk^{2}\|u\|^{2} \\ \leq&c_{\ast}-\frac{ab}{4}s^{3}-\frac{1}{24}b^{3}s^{6}- \frac {as\sqrt{b^{2}s^{4}+4(a+b\|u\|^{2})s}}{6} \\ &{}-\frac{b^{2}s^{4}\sqrt{b^{2}s^{4} +4(a+b\|u\|^{2})s}}{24} \\ &{}-\frac{(3b^{2}s^{3}+bs\sqrt{b^{2}s^{4}+4(a+b\|u\|^{2})s})\|u\| ^{2}}{24}-\frac{bk^{2}\|u\|^{2}}{4}, \end{aligned}$$

since

$$c_{\ast}< \biggl[\frac{(b+\lambda c)^{2}}{12}s^{4}+\frac{a}{3}s \biggr]\frac {(b+\lambda c)s^{2}+\sqrt{(b+\lambda c)s^{4}+4as}}{2}+\frac {ab+4\lambda ac}{12}s^{3}, $$

for λ small enough, \(\lambda\in(0,\lambda^{\ast})\), such that

$$\begin{aligned}& c_{\ast} -\biggl(\frac{b^{2}}{12}s^{4}+\frac{a}{3}s \biggr)\frac{bs^{2}+\sqrt {bs^{4}+4as}}{2}+\frac{ab}{12}s^{3}\leq0, \\& I(u) \leq c_{\ast}-\biggl(\frac{b^{2}}{12}s^{4}+ \frac{a}{3}s\biggr)\frac {bs^{2}+\sqrt{bs^{4}+4as}}{2}+\frac{ab}{12}s^{3}- \frac{bk^{2}\|u\| ^{2}}{4} \\& \hphantom{I(u)}\leq-\frac{bk^{2}\|u\|^{2}}{4} , \end{aligned}$$
(3.21)

we obtain from (3.17) and (3.21) a contradiction.

So we obtain \(\|u_{n}-u\|\rightarrow0\). The functional I possesses the mountain pass geometry, on combining Lemma 3.1. Hence u is a weak solution of the problem (1.1), we have \(\langle I'(u),u^{-}\rangle=0\), where \(u^{-}=\min\{u,0\}\). Then \(u\geq0\) and \(u\not\equiv0\), for \(c_{\ast}>0\). By the strong maximum principle we see that u is a positive solution of problem (1.1). □

Proof of Corollary 1.2

The proof is similar to Theorem 1.1, here we omit it. □