I want to show the origin of the dynamical system
\begin{align} \dot{x}_1 &= -2x_1+x_2+x_1^3x_2^2\\ \dot{x}_2 &= -x_1-2x_2+x_1^2x_2^3 \end{align}
is asymptotically stable over an invariant set $D\subset\mathbb{R}^2$. We can write this system as
$$\dot{x} = Ax + g(x) = \begin{pmatrix}-2&1\\-1&-2\end{pmatrix}x + x_1^2x_2^2x.$$
Where $x=(x_1,x_2)$. Using the quadratic form $V(x)=x^TPx$ with
$$P = \begin{pmatrix}1/4 & 0\\0& 1/4\end{pmatrix}\implies \lambda_{min}(P) = 1/4, \,\, \Vert P\Vert = \lambda_{max}(P)=1/4$$
Where $P$ is given by the formula $A^TP+PA=-I$. It can be shown that $\dot{V}$ may be written as
\begin{align} \dot{V}(x) &= -\vert x\vert^2 + 2x^TPg(x)\\ &\le -\vert x\vert^2 + 2\vert x\vert \Vert P\Vert \vert g(x)\vert \end{align}
Now note that
$$\frac{\vert g(x)\vert}{\vert x\vert} = x_1^2x_2^2 \le \frac{1}{2}(x_1^2+x_2^2)^2 = \frac{1}{2}\vert x\vert^4<\varepsilon \iff \vert x\vert < (2\varepsilon)^{1/4}$$
Thus
$$\dot{V}(x)< -\vert x\vert^2 + 2\varepsilon\vert x\vert^2 \Vert P\Vert = -\vert x\vert^2[1-2\varepsilon\Vert P\Vert]<0\iff \varepsilon<1/(2\Vert P\Vert)$$
Recall that $P$ is positive definite so
$$\lambda_{min}(P)\vert x\vert^2 \le x^TPx = V(x)$$
We also have that $\vert x\vert < (2\varepsilon)^{1/4}<1/\Vert P\Vert^{1/4}$ so
$$\lambda_{min}(P)\vert x\vert^2 < \lambda_{min}(P)/\Vert P\Vert^{1/2}$$
Since there is a case where $V(x)=\lambda_{min}(P)\vert x\vert^2$ we have that
$$\lambda_{min}(P)\vert x\vert^2 \le V(x) < \lambda_{min}(P)/\Vert P\Vert^{1/2}$$
Therefore, the dynamical system is asymptotically stable on the invariant set
$$D = \left\{x\in\mathbb{R}^2\,\colon V(x) < \frac{\lambda_{min}(P)}{\Vert P\Vert^{1/2}}\right\} = \left\{x\in\mathbb{R}^2\,\colon V(x) < \frac12\right\}$$
However my lecturer has
$$D = \left\{x\in\mathbb{R}^2\,\colon V(x) < 2\right\}$$
What have I done wrong?
This might be due to some confusion in the definition of $V$ since $x=\left(\frac32,\frac32\right)$ is not in the invariant set of this dynamical system although, with your choice of $P$, $x^TPx=\frac14|x|^2=\frac98<2$. Note that $P=\frac14I$ hence $x^TPx$ is just a fancy way of writing $\frac14x^Tx=\frac14|x|^2$. On the other hand, the (more natural) choice $V(x)=|x|^2$ yields your lecturer's answer.
In support of this explanation, note that $D=\{x\in\mathbb R^2\mid x^TPx<\frac12\}=\{x\in\mathbb R^2\mid |x|^2<2\}$ is indeed an invariant domain.