The following nonlinear system has a fixed point in the origin. I want to know if this fixed point is stable:
\begin{align*} &\dot{\alpha}=\alpha^2-2\beta(\alpha+\beta)\\ &\dot{\beta}=\beta^2-2\alpha(\alpha+\beta) \end{align*}
Note that the linearization of the system leads to $\dot{\alpha}=\dot{\beta}=0$ to first order, so this method is useless. To find out the stability, I wrote this equations as:
\begin{align*} \dot{\alpha}=\left(\alpha,\beta\right) \cdot A\cdot \left(\begin{array}{c} \alpha\\ \beta \end{array}\right),\quad A=\left(\begin{array}{cc} 1 & -1\\ -1 & -2 \end{array}\right) \end{align*}
\begin{align*} \dot{\beta}=\left(\alpha,\beta\right) \cdot B\cdot \left(\begin{array}{c} \alpha\\ \beta \end{array}\right),\quad B=\left(\begin{array}{cc} -2 & -1\\ -1 & 1 \end{array}\right) \end{align*}
Since $A$ and $B$ are real and symmetric, they are diagonalizable, with real eigenvalues. It turns out that $A$ and $B$ have one eigenvalue positive and the other negative (in fact, $A$ and $B$ have the same eigenvalues), explicitly:
\begin{align*} &\text{Eigenvalues of $A$:}\quad \frac{1}{2} \left(-1-\sqrt{13}\right),\frac{1}{2} \left(\sqrt{13}-1\right)\\ &\text{Eigenvalues of $B$:}\quad \text{same of $A$} \end{align*} If, for example, the eigenvalues of $A$ were both positive, then $\dot{\alpha}$ would be positive for all points except origin, and then the origin would be unstable (because trajectories starting out of the origin would increase their $\alpha$'s to infinity). But when both $\dot{\alpha}$ and $\dot{\beta}$ are positive or negative depending on the point $(\alpha,\beta)$, as in this case, how can you determine the stability of the fixed point?
EDIT: The phase portrait is:
This is mostly a recap of the observations made in the comments, plus some more analysis, because I think it's a nice problem to analyse.
First, both the functional form of the system and the reflection symmetry (see also the phase plane) suggest it's a good idea to introduce $x = \alpha+\beta$, $y = \alpha-\beta$, to obtain \begin{align} \dot{x} &= \frac{1}{2}(y^2 - 3 x^2), \tag{1a}\\ \dot{y} &= 3 x y.\tag{1b} \end{align} We can simplify system (1) somewhat by rescaling $y \to \sqrt{3} y$ and $t \to \frac{1}{3} t$, yielding \begin{align} \dot{x} &= \frac{1}{2}(y^2 - x^2), \tag{2a}\\ \dot{y} &= x y.\tag{2b} \end{align} The phase plane of system (2) looks like this:
What's immediately obvious, is that this phase plane is highly symmetric. It seems to be invariant under rotation over an angle of $\frac{2 \pi}{3}$ (= 120 deg), and rotation over an angle of $\frac{2 \pi}{6}$ seems to keep the shape of the orbits invariant, but changes their flow direction. You can check that both of these observations are indeed correct by considering \begin{equation} \begin{pmatrix} \xi_1 \\ \eta_1 \end{pmatrix} := R(\frac{2\pi}{3}) \begin{pmatrix} x \\ y \end{pmatrix}\quad \text{and} \quad \begin{pmatrix} \xi_2 \\ \eta_2 \end{pmatrix} := R(\frac{2\pi}{6}) \begin{pmatrix} x \\ y \end{pmatrix}, \end{equation} with \begin{equation} R(\theta) = \begin{pmatrix} \cos \theta & \sin \theta \\ -\sin \theta & \cos \theta \end{pmatrix} \end{equation} is the matrix that rotates a vector about the origin over angle $\theta$. Substituting $\xi{1,2}$ and $\eta_{1,2}$ in system (2), you obtain \begin{align} \dot{\xi_1} &= \frac{1}{2}(\eta_1^2 - \xi_1^2), \\ \dot{\eta_1} &= \xi_1 \eta_1, \end{align} and \begin{align} \dot{\xi_2} &= -\frac{1}{2}(\eta_2^2 - \xi_2^2), \\ \dot{\eta_2} &= -\xi_2 \eta_2, \end{align} which implies the above observations.
It seems a good idea to 'factor out' this rotational symmetry present in system (2). Introducing polar coordinates $x = r \cos \theta$, $y = r \sin \theta$, we obtain the dynamical system \begin{align} \dot{r} &= -\frac{1}{2} r^2 \cos(3\theta),\tag{3a}\\ \dot{\theta} &= \frac{1}{2} r \sin(3\theta),\tag{3b} \end{align} which is indeed invariant under $\theta \to \theta + \frac{2 \pi}{3}$. Now, we rewrite the above in terms of the new angle $\phi := 3 \theta$ to obtain \begin{align} \dot{r} &= -\frac{1}{2} r^2 \cos(\phi), \tag{4a}\\ \dot{\phi} &= \frac{3}{2} r \sin(\phi). \tag{4b} \end{align} Now, the nice thing is that we can reinterpret system (4) as being the `polar representation' of some Cartesian system. That is, if we introduce $X$ and $Y$ by $X := r \cos \phi$ and $Y := r \sin \phi$, we can rewrite system (4) in terms of $X$ and $Y$ to obtain \begin{align} \dot{X} &= -\frac{1}{2}(X^2 + 3 Y^2), \tag{5a}\\ \dot{Y} &= X Y. \tag{5b} \end{align} The phase plane of system (5) looks as we would have expected:
Moreover, system (5) turns out to be Hamiltonian, i.e. of the form \begin{align} \dot{X} &= \frac{\partial H}{\partial Y},\\ \dot{Y} &= -\frac{\partial H}{\partial X}, \end{align} with \begin{equation} H(X,Y) = -\frac{1}{2} Y(X^2 + Y^2). \end{equation} Therefore, orbits lie on level sets of $H$, i.e. those curves where $H = E$ (= constant), which can be used to express $X$ in terms of $Y$ as \begin{equation} X = \pm \sqrt{-\frac{2 E + Y^3}{Y}}. \end{equation}
As a final remark, the instability of the origin can be derived from system (5) as follows. Consider the horizontal line $\ell = \left\{ (X,Y) \vert Y=0 \right\}$, i.e. the $X$-axis. Substituting $Y=0$ in system (5), we obtain $\dot{Y} = 0$; therefore, the line $\ell$ is invariant under the flow. In other words, every point on $\ell$ stays on $\ell$ for all time.
It so happens that the origin $(0,0)$ lies on the line $\ell$. Taking a point $(-\epsilon,0) \in \ell$ arbitrarily close to (and directly to the left of) the origin, we see that this point will flow away from the origin, because the flow on $\ell$ is given by $\dot{X} = -\frac{1}{2} X^2$. This holds for all $\epsilon > 0$; hence, we have found an unstable flow direction at the origin, spanned by $(-1,0)$. Therefore, the origin is a (nonlinearly) unstable fixed point of system (5), and in extension of system (1).