
I have tried numerous functions for $V(x)$ but I am having trouble finding one that tells me about the stability at the equilibrium point. Any ideas of one that may work?

I have tried numerous functions for $V(x)$ but I am having trouble finding one that tells me about the stability at the equilibrium point. Any ideas of one that may work?
try the first one again (V(x) = ax^2 +by^2), except let a=b after you multiple V'(x)f(x).
you will see that one pair of terms cancels. pull out the -2b and then 3 of the terms will be a difference of squares. factor those terms and you will be left with one term that is squared and one power that is to the 4th (so they are both positive). Since this positive term is multiplied by a -2, the whole thing is negative and QED.