Given $A$ is an $n\times n$ matrix. Find the stability of the origin of the linear system $\dot{x} = Ax$ if the coefficient matrix satisfies $A^2 = 0$. How about that of the origin when $A^2 = A$?
My attempt: Assume $\lambda$ is an eigenvalue of $A$ corresponding to an eigenvector $y$ ($y\neq 0$). We will show that $\lambda$ must be $0$ when $A^2=0$ as follows (note that we are not given the assumption that $A$ is invertible, so we cannot have $A=0$).
We have: $Ay = \lambda y$, so $A^2y = \lambda Ay$. Since $A^2 = 0$, we get: $\lambda = 0$ is the only choice (since $Ay\neq 0$). Thus, the only eigenvalue of $A$ is $0$, but $0$ is not semi-simple in this case, unless $A$ has $n$ LI columns. Thus, the origin is unstable.
For $A^2 = A$, using the same trick, we get: $\lambda = 1 > 0$ is one of an eigenvalues of $A$, so the origin is unstable as well (Q.E.D)
My question: Is what I did here correct? If not, where's my mistake?
One should distinguish between the cases $A=0$ and $A\ne 0$. When $A=0$, the origin is Lyapunov stable but not asymptotically stable. If $A\ne 0$, either of the assumptions leads to unstable origin; your conclusion is correct in this case. Also, you did not make it clear what kind of stability is under consideration.
Idempotent matrix: $A^2=A$
Since $A\ne 0$, then there is $x\in\mathbb{R}^n$ such that $Ax\ne 0$: since $AAx=Ax$, this gives an eigenvector with eigenvalue $1$. The origin is unstable in this case.
You don't have to prove that $1$ is the only eigenvalue: finding one eigenvalue with positive real part is enough.
Nilpotent matrix: $A^k=0$ (you have $k=2$)
Here, $0$ is the only eigenvalue, since $Ax=\lambda x$ implies $0 = A^kx = \lambda^k x$. This rules out asymptotic stability, but leaves the question of Lyapunov stability. A quick way to settle this question is to notice that $$\frac{d^k}{dt^k} x = \frac{d^{k-1}}{dt^{k-1}}Ax = A \frac{d^{k-1}}{dt^{k-1}}x = \dots = A^kx = 0$$ meaning that every trajectory is a polynomial in $t$. Hence, every nonconstant trajectory (i.e., with $Ax\ne 0$) is unbounded, proving that the origin is unstable.