Let $A$ be a $3×4$ matrix. Estimate $\det(A'A)$ and $\det(AA')$.
I would first assume that $A$ has rank $3$. Then $A'A$ would be a $4\times 4$ matrix with rank $3$ and therefore it would have zero determinant. If the rank equals $4$ I can't say anything about the determinant only that it is not zero.
I can't estimate $\det(AA')$
Am I right?
On
If $A$ is $3\times4$ with rank $3$, then (considering matrices as maps on column vectors) $A$ is surjective and $A'$ is injective, thus $\dim\ image(A'A) = 3 = rank(A'A)$. But since $A'A$ is $4\times4$, we must have $\det(A'A) = 0$, as you already noted. Now, from $rank\ A' = 3$ we get $ker\ A' = 0$, i.e. $x \neq 0 \Rightarrow A'x \neq 0$. Now assume there is a $y\neq0$ such that $AA'y = 0$. This implies $0 = y'AA'y = (A'y)'(A'y) = \|A'y\|^2 > 0$ since $A'y\neq0$ since $y\neq0$. This is a contradiction. This means that $ker\ AA' = 0$. This together with the fact that $AA'$ is square $3\times3$ implies that $AA'$ has full rank, namely $3$. So $AA'$ is invertible, and we must have $\det(AA') \neq 0$.
The rank of $A$ is at most $3$; if it is $3$, then $AA'$ has the same rank, so it's invertible and will have nonzero determinant. Otherwise the determinant is $0$.
The rank of $A'A$ is at most $3$, so this matrix always has zero determinant.
What the determinant of $AA'$ is, in case $\operatorname{rank}A=3$ cannot be “estimated”: just take $$ A=\begin{bmatrix} a & 0 & 0 & 0\\ 0 & 1 & 0 & 0\\ 0 & 0 & 1 & 0 \end{bmatrix} $$ and $\det(AA')=a^2$. So any positive number can be the determinant.
Note however that all eigenvalues of $AA'$ are real (the matrix is symmetric) and non negative: indeed, if $AA'v=\lambda v$, then $$ 0\le(A'v)'(A'v)=v'AA'v=v'(\lambda v)=\lambda v'v $$ and therefore $\det(AA')\ge0$.