We have been given matrix:
$$A =
\begin{pmatrix}
a& b& c &d \\
b &−a& d& −c\\
c& −d &−a& b \\
d &c& −b& −a\\
\end{pmatrix}
$$
...and have been asked to calculate $\det(A)$ using $AA^T$.
We see that:
$$AA^T=\begin{pmatrix} a^2+ b^2+ c^2+ d^2& 0& 0&0\\ 0 &a^2+ b^2+ c^2+ d^2& 0& 0\\ 0& 0 &a^2+ b^2+ c^2+ d^2& b \\ 0 &0& 0& a^2+ b^2+ c^2+ d^2\\ \end{pmatrix} $$
So, $\det(AA^T)= (a^2+ b^2+ c^2+ d^2)^4$
Now which should I choose: $(a^2+ b^2+ c^2+ d^2)^2 $ or $ -(a^2+ b^2+ c^2+ d^2)^2$ ?
Please explain me which one and why.
This might be an overkill but it is a useful observation nonetheless. You have a matrix $A = A(a,b,c,d)$ which depends on four real parameters and you computed $\det(A)^2$. From your computation, it is clear that $\det(A) = 0$ iff $a = b = c = d = 0$. Set $U = \mathbb{R}^4 \setminus \{ (0,0,0,0) \}$. If $(a,b,c,d) \in U$ then $\det(A) \neq 0$ so either $\det(A) > 0$ or $\det(A) < 0$. However, the set $U$ is connected and the function $U \rightarrow \mathbb{R}$ given by $(a,b,c,d) \mapsto \det(A(a,b,c,d))$ is continuous so we must have $\det(A) > 0$ for all $(a,b,c,d) \in U$ or $\det(A) < 0$ for all $(a,b,c,d) \in U$ and we can check which by plugging in specific values for $a,b,c,d$. For example, when $b = c = d = 0$ we have
$$ \det(A) = a(-a)^3 = -a^4 $$
so $\det(A) < 0$ if $a \neq 0$ and hence
$$ \det(A(a,b,c,d)) = -(a^2 + b^2 + c^2 + d^2)^2 $$
for all $(a,b,c,d) \in U$.