$A=\begin{bmatrix} a&1&0\\ 1&b&d\\ 1&b&c \end{bmatrix}, B=\begin{bmatrix} a&1&1\\ 0&d&c\\ f&g&h\end{bmatrix}, U=\begin{bmatrix} f\\ g\\ h \end{bmatrix} U=\begin{bmatrix} a^2\\ 0\\ 0 \end{bmatrix}, X=\begin{bmatrix} x\\ y\\ z \end{bmatrix}$
If $AX = U$ has infinitely many solutions, then prove that $BX=V$ cannot have a unique solution. If further $afd\ne0$, then prove that $BX=V$ has no solution.
My attempt is as follows:-
If $AX=U$ has infinitely many solutions, then $|A|=0$
$$\begin{vmatrix} a&1&0\\ 1&b&d\\ 1&b&c \end{vmatrix}=0$$
$$a(bc-bd)-(c-d)=0$$ $$ab(c-d)-(c-d)=0$$ $$(c-d)(ab-1)=0$$
So either $c=d$ or $ab=1$
Case $1$: $c=d$
$$AX=U$$
Taking $adj(A)$ on both sides
$$adj(A)AX=adj(A)U$$ $$OX=adj(A)U$$
$$adj(A)=\begin{bmatrix} bc-bd&-c&d\\ -(c-d)&ac&-ad\\ 0&-(ab-1)&ab-1 \end{bmatrix}$$
$$adj(A)U=0$$ $$\begin{bmatrix} (bc-bd)f-cg+dh\\ -(c-d)f+acg-adh\\ -(ab-1)g+(ab-1)h \end{bmatrix}=\begin{bmatrix} 0\\ 0\\ 0 \end{bmatrix}$$
As $c-d=0$
$$\begin{bmatrix} -cg+ch\\ acg-ach\\ -(ab-1)g+(ab-1)h \end{bmatrix}=\begin{bmatrix} 0\\ 0\\ 0 \end{bmatrix}$$ $$g=h$$
Now let's prove $BX=V$ cannot have a unique solution
$$|B|=a(dh-cg)-(0-cf)+(0-df)$$ $$|B|=0$$
$$BX=V$$
Taking $adj(B)$ on both sides
$$OX=adj(B)V$$
$$adj(B)=\begin{bmatrix}\ dh-gc&g-h&c-d\\ cf&ah-f&-ac\\ -df&-(ag-f)&ad \end{bmatrix}$$
$$adj(B)V=\begin{bmatrix} a^2(dh-gc)\\ a^2cf\\ -a^2df \end{bmatrix}$$
$$adj(B)V=a\begin{bmatrix} a(dh-gc)\\ acf\\ -adf \end{bmatrix}$$
So if $afd$ is not equal to zero, then there will be no solution otherwise it will have infinitely many solutions.
Case $2$: $ab-1=0$
$$adj(A)U=0$$ $$\begin{bmatrix} (bc-bd)f-cg+dh\\ -(c-d)f+acg-adh\\ -(ab-1)g+(ab-1)h \end{bmatrix}=\begin{bmatrix} 0\\ 0\\ 0 \end{bmatrix}$$ $$b(c-d)f-cg+dh=0\tag{1}$$ $$-(c-d)f+acg-adh=0\tag{2}$$
Multiplying first equation by $-a$
$$-(c-d)f+acg-adh=0$$
So first and second equation are same.
Now let's prove $BX=V$ cannot have a unique solution
$$|B|=a(dh-cg)-(0-cf)+(0-df)$$ $$|B|=a(dh-cg)+cf-df$$
$$|B|=-(c-d)f+cf-df$$ $$|B|=0$$
Taking $adj(B)$ on both sides
$$OX=adj(B)V$$
$$adj(B)V=a\begin{bmatrix} a(dh-gc)\\ acf\\ -adf \end{bmatrix}$$
So yes here also if $afd\ne0$, then $BX=V$ will have no consistent solution.
My solution went very long, any easy to solve this?
Theorem 1. For any real square matrix $W \in \mathbb{R}^{n \times n}$ for any vector $y \in \mathbb{R}^n$, the equation $Wx=y$ has a unique solution if and only if $W$ is invertible.
So, it's enough to prove that $B$ is not invertible. First consider the case when $U=0$, i.e. $f=g=h=0$. In that case $B$ is obviously not invertible.
Now assume $U \neq 0$. Since $AX=U$ has infinitely many solutions, $A$ is not invertible. Now, consider the span of the two vectors $\begin{bmatrix} a \\ 1 \\ 1 \end{bmatrix}, \begin{bmatrix} 0 \\ d \\ c \end{bmatrix}$. If it's not a 2-dimensional vector subspace, then it's 1-dimensional, in which case $B$ is not invertible. So assume it's 2-dimensional. Then, the span of these 2 vectors is the same as the column span of $A$. Therefore, the vector $U$ is in the span of these 2 vectors. But now remember that the rows of the matrix $B$ consist of those 2 vectors and also of the vector $U$. Now we know that these rows are linearly dependent, therefore the matrix $B$ is not invertible. QED