For $2X2$: $$ A:\\ \begin{bmatrix} a & b \\ c & d \end{bmatrix} $$ $$ \operatorname{Adj}(A):\\ \begin{bmatrix} d & -c \\ -b & a \end{bmatrix} $$ So the statement is true.
The problem comes when it's $3X3$: $$ \operatorname{Adj}(A):\\ \begin{bmatrix} \left |\begin{matrix} a_{22} & a_{23} \\ a_{32} & a_{33} \end{matrix} \right | & 0 & 0 \\ 0 & \left |\begin{matrix} a_{11} & a_{13} \\ a_{31} & a_{33} \end{matrix} \right |& 0 \\ 0 & 0 & \left |\begin{matrix} a_{11} & a_{12} \\ a_{21} & a_{22} \end{matrix} \right | \end{bmatrix} $$ I'm not sure this implies that A is diagonlizable. I've done some examples and the statement was true but I can't prove the general case.
EDIT:
I just realized that I confused diagonal with diagonalizable so you can ignore all my steps.
The result is not true. If $A \in M_n(\mathbb{F})$ is a non-diagonalizable matrix with $\operatorname{rank} A \leq n - 2$ then all the $(n-1) \times (n-1)$ minors are zero and so $\operatorname{Adj}(A) = 0$ is diagonalizable. For a concrete counterexample, you can take
$$ A = \begin{pmatrix} 0 & 1 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \end{pmatrix}. $$