Usually you can note some nice structure in the problem which enables construction of a nice Lyapunov function. But this one is just a monster. Maybe there is a trick I've missed?
Investigate the stability of the origin of the following system.
$$x' = yx - 5x^3 \\ y' = x^2 - 5y$$
Attempt: First note that the are no other fixed points. Also note that linearization will fail at the origin.
We try some Lyapunov function $V(x,y) = G(x) + H(y)$ which gives
$$V' = g(x)xy - g(x)5x^3 + h(y)x^2-h(y)5y$$
I can not figure out what kind of function would make $V'$ definite in either direction, due to the differences in order. I should also note that I've already caught a lot of sloppy errors in these exam questions, making the problems either impossible to solve or something other than the posted solution. This particular problem has no solutions posted, however.