Let $M$ and $N$ be two $3\times 3$ matrices such that $MN=NM$, Further if $M\neq N^2$ and $M^2=N^4$ then
(A) Determinant of $(M^2+MN^2)$ is $0$
(B) There is a non-zero $3\times 3$ matrix $U$ such that $(M^2+MN^2)U$ is the zero matrix
(C) Determinant of $(M^2+MN^2) \geq1$
(D) For a $3\times 3$ Matrix $U$, if $(M^2+MN^2)U$ equals the zero matrix then $U$ is the zero matrix.
My Thinking:
$(M-N^2)(M+N^2)=M^2-N^4+MN^2-N^2M$
$\implies$ $(M-N^2)(M+N^2)=MNN-NNM=NMN-NMN=0$ (Here zero denotes Null matrix).
Case $(1)$ $M+N^2 \neq0$ and $M-N^2= 0$ (Here zero is Null matrix)
But it is given that $M\neq N^2$
So Case $1$ Can't be true.
Case$(2)$ $M-N^2\neq 0$ and $M+N^2\neq 0$
But we know that $|M-N^2||M+N^2|=0$
$\implies$ $|M+N^2||M-N^2|= 0$
So
Subcase $(1)$ of case $(2)$ $|M+N^2|=0$ and $|M-N^2|\neq 0$
Subcase $(2)$ of case $(2)$ $|M+N^2|\neq 0$ and $|M-N^2|=0$
But when $|M+N^2|\neq0$ then we can take inverse of this.
$\implies$
$0=(M+N^2)(M-N^2)$
$\implies$
$0=(M+N^2)^{-1}(M+N^2)(M-N^2)$
$\implies$
$0= M-N^2$ i.e. subcase $(2)$ is false.
$\implies$ Result of case $(2)$ is $|M+N^2|=0$
Case $(3)$
$M+N^2=0$ and $M-N^2\neq 0$
Result of case $(3)$ is $|M+N^2|=0$
If above is true then Matrix $(M+N^2)=0$
From All three cases $|M+N^2|=0$.
So Option $(A)$ is True.
My Doubt is option (B)
I think this statement should be false because $(M^2+MN^2)U$ need not be zero always.
This question was asked in Indian Exam JEE Advanced Paper $1$ of Year $2014$ https://www.jeeadv.ac.in/archive.html
Linked Question Finding the value of the determinant $|M^2+MN^2|$ and determining whether $U$ is a zero matrix or not
As mentioned in the comments by @SarveshRavichandranIyer, you only need to prove the existence of one matrix $U$ such that $(M^2+MN^2)U = 0$, not that it holds for any $U$.
You've already proven that (A) is true, i.e. the determinant $|M^2+MN^2| = 0$. For the sake of simplicity, let's set $A := M^2+MN^2$. We know that $|A| = 0$ and have to prove that there exists some matrix $U$ such that $AU = 0$.
Because $|A| = 0$, the columns of $A$ are linearly dependent. Let's say that $c_1 = x\, c_2 + y\, c_3$, where $c_i$ is the $i$-th column of $A$. Now let $$U := \begin{pmatrix}1&1&1\\-x&-x&-x\\-y&-y&-y\\\end{pmatrix}.$$ It's easy to verify that $AU = 0$, but $U$ is a non-zero matrix.