I have verified using the total energy function that around $(0,0)$ the nonlinear system is stable:
\begin{align}\dot x&=y \\ \dot y &= -x + \beta x^3 - \gamma y \end{align}
where $\beta, \gamma > 0$ are constants. However, I have been trying to find a suitable strict Lyapunov function $V$ but from the expressions I have come up with so far, I cannot definitively deduce that the derivative is less than zero (If I use linearization, then $(0,0)$ is an asymptotically stable sink. I have tried the classical $V = x^2 + y^2$ and tried changing up the coefficients and exponents so that I can have some cancellations of the odd terms. It has been several hours now and still no luck. Any hints will be much appreciated. Moreover, the total energy function that I use for this system is $H(x,y) = \frac{y^2}{2} + \frac{x^2}{2} - \frac{\beta x^4}{4}$.
A Lyapunov function can be obtained from the linearized system. The linearized system is $$\dot x= y,$$ $$\dot y=-x - \gamma y$$ or $$ \left(\begin{array}{c}\dot x\\\dot y\end{array}\right)= A \left(\begin{array}{c}x\\y\end{array}\right), $$ where $$ A=\left(\begin{array}{rr}0&1\\-1&-\gamma \end{array}\right). $$ Solving the Lyapunov equation $A^TP+PA=-Q$, where $P$ is an unknown matrix, $Q$ is any positive-definite matrix, one can obtain the solution $P$, which can be used to construct the Lyapunov function $$ V(x,y)=\left(\begin{array}{cc}x&y\end{array}\right) P \left(\begin{array}{r}x\\y \end{array}\right). $$ For example, $$ Q=\left(\begin{array}{rr}2&0\\0&2 \end{array}\right) $$ gives us $$ P=\left(\begin{array}{cc}\frac2{\gamma}+\gamma&1\\1&\frac2{\gamma} \end{array}\right). $$ Thus, one of the possible Lyapunov functions is $$ V(x,y)=\left( \frac2{\gamma}+\gamma\right)x^2+2xy+\frac2{\gamma}y^2. $$