Let $\mathbf B$ denote an $n \times n$ matrix with $r\equiv\operatorname{rank}(\mathbf B)$. I need to prove the following conjecture:
If $r \le n - 2$, then there exists a polynomial matrix $\mathbf P(x)$ such that $$ \operatorname{Adj} (\mathbf I_n x-\mathbf B)= x^{n-r-1}\mathbf P(x). $$
We need the following results:
3 implies the $i,j$th element of $\text{Adj}(xI-A)$ is $$ \begin{align} \left|xI-A+e_je_i^{\small T}\right| - \left| xI - A \right| = - x^{n-1}(U_1 - S_1) + x^{n-2}(U_2 - S_2) + \dots + (-1)^n(U_n - S_n) \end{align} $$ where for each $k$ $U_k$ is the sum of all $k \times k$ principal minors of $A-e_je_i^{\small T}$ and $S_k$ is the sum of all $k \times k$ principal minors of $A$.
Since $\texttt{rank}(A) = r$ we have $S_{r+1}=\dots=S_n = 0.$
We will now show $U_{r+2}=\dots=U_n=0.$
Let $M$ be a $k \times k$ principal submatrix of $A-e_je_i^{\small T}$ and let $N$ be the principal submatrix of $A$ corresponding to the same indices which determine $M$. $M$ and $N$ can differ in at most one entry by exactly 1. So if $M$ and $N$ differ at the $h,g$th position and $N = \left[ \begin{matrix} n_1 & n_2 & \dots & n_k \end{matrix} \right]$ then $M = \left[ \begin{matrix} n_1 & \dots & n_{g-1} & n_g - e_h & \dots & n_{g+1} & \dots & n_k \end{matrix} \right]$ which implies $\left|M\right| = \left|N\right| - \left| \begin{matrix} n_1 & \dots & n_{g-1} & e_h & n_{g+1} & \dots & n_k \end{matrix} \right|.$ We can see that the second term in the previous sum is a $(k-1)\times(k-1)$ minor of $A$ (upto a sign). And since $\texttt{rank}(A)=r$ this implies if $k \geq r+2$ then $|M|=0$ and so $U_{r+2}=\dots=U_n=0$.
So the $i,j$th element of $\text{Adj}(A)$ is of the form $-x^{n-1}(U_1-S_1) + \dots \pm x^{n-r+1}(U_{r+1}-S_{r+1})$ and we are done.
Finally $1$ can be proved by noting $f(x) = |xI - A|$ is a polynomial in $x$ and evaluating the coefficient of $x^k$ as $f^{k}(0)/k!$.
2 can proved as follows. First assume $A$ is invertible then $$ \begin{align} |A+uv^T| &= |A||I + (A^{-1}u)v^T|,\\ &= |A|(1+v^TA^{-1}u), \\ &= |A| + v^T|A|A^{-1}u,\\ &= |A| +v^T\text{Adj}(A)u. \end{align}$$
Since $|A+uv^T|$ and $|A| +v^T\text{Adj}(A)u$ are continuous functions of $A$ since they are both polynomials in the entries of $A$ and non-singular matrices are dense, given any singular matrix $A$ we can find a sequence of non-singular matrices $A_n$ converging to $A$ and result follows from $|A+uv^T| = \lim |A_n+uv^T| = \lim |A_n| +v^T\text{Adj}(A_n)u = |A| +v^T\text{Adj}(A)u.$