I am working on mountain-pass like theorems for the problems
$$ - u_{xx} - a u = \pm |u|u+|u|^2u , \ x \in (a,b), \quad u(a)=u(b) = 0$$
where $a \in L^\infty((a,b))$ is positive (I take the one dimensional case for simplicity on Sobolev immersions). Thus the associated bilinear form $a(u,v)$ becomes non coercive. Of course, I have the following functional
$$ J(u) = \frac 1 2 \int_a^b (u_x^2 - au^2) - \int_a^b F(u) $$
where $F(u) = \int_a^b (\mp \frac 1 3 |u|^3 + \frac 1 4 |u|^4)$.
My problem is to very the Palais-Smale (PS) compacteness conditions for this functional.
I have reviewed Evans' Partial Differential Equations with hardly any luck (only considers an example on the coercive case). I have also reviewed a paper by Radulescu in 2012 on this matter, which solves the problem for $a = \lambda$ is constant and $f(u)$ strictly increasing and onto, so $|u|u+|u|^2u$ would be solved in a particular case. This last paper applies the mountain pass theorem to $v=f(u)$ allowing for the existence of $(-\Delta + \lambda I)^{-1}$ (which is hardly possible if $a$ is not constant).
I found a theorem which takes into account the spectral decomposition of the bilinear form, and allows for a fairly straightforward proof if the (PS) conditions are verified.
Does anybody know how to show (PS) or of papers where they tackle with (PS) and noncoercive operator?
Thank you ever so much, D
I will do here some calculations, which may help you to get a full understanding of the problem. Because you had alred used $a$ in the domain $(a,b)$, I will use $\alpha$ for the function in $L^\infty(a,b)$.
Assume for instance that $u_n\in H_0^1(a,b)$ satisfies $$J(u_n)\le M,\ ~~ J'(u_n)\to 0\tag{1}$$
We have two cases:
I - We take the minus sign on the cubic term.
In this case we have that for $n$ big enough,
\begin{eqnarray} M+\|u_n\|_{1,2} &\ge& J(u_n) -\frac{1}{4}\langle J'(u_n),u_n\rangle \nonumber \\ &=& \frac{1}{4}\|u_n\|^2_{1,2}-\frac{1}{4}\int \alpha u_n^2+\frac{1}{12}\int |u_n|^3 \tag{2} \end{eqnarray}
Therefore, we obtain from $(2)$
\begin{eqnarray} M+\|u_n\|_{1,2} &\ge& \frac{1}{4}\|u_n\|^2_{1,2}-\frac{1}{4}\int \alpha u_n^2 \nonumber \\ &\ge& \frac{1}{4}\left(1-\frac{\|\alpha\|_\infty}{\lambda_1}\right)\|u_n\|_{1,2}^2 \tag{3} \end{eqnarray}
where in $(3)$ we have use Poincare's inequality and $\lambda_1$ is the first eigenvalues of $(H_0^1(a,b),-\Delta)$. If $\|\alpha\|_\infty<\lambda_1$, we conclude from $(3)$ that $u_n$ is bounded in $H_0^1$. I will leave to you the task to conclude PS from here (in the case $\|\alpha\|_\infty<\lambda_1$).
II - We take plus sign.
As in case I, we have for big $n$
\begin{eqnarray} M+\|u_n\|_{1,2} &\ge& J(u_n) -\frac{1}{3}\langle J'(u_n),u_n\rangle \nonumber \\ &=& \frac{1}{6}\|u_n\|^2_{1,2}-\frac{1}{6}\int \alpha u_n^2+\frac{1}{12}\int |u_n|^4 \tag{2} \end{eqnarray}
As in case I, we can now conclude that for $\|\alpha\|_\infty<\lambda_1$, the sequence $u_n$ is bounded and then PS follows.
Remark: I think that the condition $\|\alpha\|_\infty<\lambda_1$ is necessary.