I'm fairly certain I have this worked out but I want to make sure of it.
Let $A$ and $B$ be $5\times 5$ matrices and $P(A)$ be the vector space of all solutions to the homogeneous system $Ax=0$ . Given that $rank(B)=3$, then:
$rank(AB)=3 \Leftrightarrow$ A is invertible.
If $AB=0$ then $dim P(A)\leq 2$
I believe the first statement is false because the rank of the product of matricial multiplication is at most the rank of the matrix on the left side of the operation. If $A$ is invertible its rank is 5, but the question specified 3, so I know it's possible that there is some $A,\ 3\leq rank(A)<5\Rightarrow rank(AB)=3$ which disproves the statement.
As for the second statement I believe it is false as well. My reasoning is as follows: if $AB=0$ that means that every column of $B$ is itself a solution to $Ax=0$, and since $rank(B)=3$ that means that the dimension of $B$'s column space is also 3. That means that $dimP(A)= 3$ or 4 if $B$ doesn't contain a column of $0$s, but either way the statement is false.
Am I correct in my assertions?
$$ A = B = \begin{bmatrix} 1 & 0 & 0 & 0 & 0\\ 0 & 1 & 0 & 0 & 0\\ 0 & 0 & 1 & 0 & 0\\ 0 & 0 & 0 & 0 & 0\\ 0 & 0 & 0 & 0 & 0\\ \end{bmatrix} $$
$$ AB = \begin{bmatrix} 1 & 0 & 1 & 0 & 0\\ 0 & 1 & 0 & 1 & 0\\ 0 & 0 & 1 & 0 & 1\\ 0 & 0 & 0 & 0 & 0\\ 0 & 0 & 0 & 0 & 0\\ \end{bmatrix} \begin{bmatrix} 1 & 0 & 0 & 0 & 0\\ 0 & 1 & 0 & 0 & 0\\ -1 & 0 & 1 & 0 & 0\\ 0 & -1 & 0 & 0 & 0\\ 0 & 0 & -1 & 0 & 0\\ \end{bmatrix} = \mathbf{0} $$
These matrices are in exactly the situation you described.