Consider a given PSD matrix $S$, which can be written as:
$$S= A A^\top$$
where $A$ is its unique PSD square root. I am computing $A$ through the EVD of $S= U DU^\top$, i.e., $A= U D^{1/2}U^\top$.
I want to fit a model to $S$ as:
\begin{equation} S= B(\theta) B(\theta)^\top \end{equation} and specifically I look after the parameter $\theta$ from which the square matrix $B$ depends.
My final goal is to use $B(\theta)$ in place of $A$. Now, I would be tempted to consider the following simpler problem:
$$A= B(\theta)$$
I know that in principle $A=B(\theta) Q $ for any orthogonal matrix $Q$. However, what I care about is not to find a $\theta$ which is equal to a true unknown parameter $\theta^\star$, rather I just want that a $\theta$ for which $B(\theta)B(\theta)^\top=S$. So my initial line of thought was to just discard the $Q$ since at the end when I multiply $B(\theta) Q Q^\top B(\theta)^\top$ the $Q$ would simply vanish.
What is wrong with this line of thought? Thank you.
Here's an example that illustrates that $S = B(\theta)B(\theta)^\top$ might have a solution when $A = B(\theta)$ doesn't. For instance, consider $$ S = \pmatrix{2 & 3\\3 & 5}, \quad A = \pmatrix{1 & 1\\1 & 2}, \quad B(\theta) = \pmatrix{1 & -1\\1 & \theta}. $$ It is clear that $B(\theta) = A$ has no solution. However, we have $B(\theta)B(\theta)^\top = S$ with $\theta = -2$.
On the other hand, there exists an orthogonal $Q$ such that $B(\theta)Q = A$. Indeed, this holds with $$ Q = \pmatrix{1&0\\0&-1}. $$
Let $S$ be $n \times n$ and let $r$ denote the (known) rank of $S$. Because of what is stated here, there exists a unique "lower-triangular" $n \times r$ matrix $L_*$ with positive diagonal entries such that $S = L_*L_*^\top$. That is, $$ L_* = \pmatrix{L_1\\L_2} $$ where $L_1$ is square and lower-triangular (with positive diagonal entries). Suppose that for all $\theta$, $B(\theta)$ has rank $r$. For any $\theta$, let $L(\theta)$ denote the $n \times r$ "lower-triangular" matrix (with positive diagonal entries) obtained from the LQ decomposition $$ B(\theta) = L(\theta) Q(\theta), $$ where $Q(\theta)$ has orthonormal rows. As with the Cholesky decomposition, the $L(\theta)$ obtained in this fashion is unique. We find that $S = B(\theta)B(\theta)^\top$ has a solution if and only if $L_* = L(\theta)$ has a solution. If $\theta_*$ is such that $L(\theta_*) = L_*$, then it immediately follows that $$ B(\theta_*) B(\theta_*)^\top = L_*L_*^\top = S. $$ Note that the LQ-decomposition is related to the QR-decomposition as follows: $$ A = LQ \iff A^\top = Q^\top L^\top, $$ where we note that $Q^\top$ has orthonormal columns and $L^\top$ is upper-triangular.
If it is desired, we can get a solution to $A = B(\theta)Q$ (with $Q$ orthogonal) as follows: Let $\tilde L_*$ denote the square matrix $[L_*\ \ 0]$. If $\tilde L_* = U \Sigma V^\top$ is a singular value decomposition, then $S = U\Sigma^2 U^\top$ and $A = U \Sigma U^\top$, which means that $A = \tilde L_* UV^\top$, or equivalently $L_* = AVU^\top$. We can add rows to $Q(\theta_*)$ to make an orthogonal matrix $Q_*$. We have $$ B(\theta_*) = \tilde L_* Q_* = (AVU^\top)Q_* = A[VU^TQ_*] \implies\\ A = B(\theta_*)Q_*^\top UV^\top. $$
Some notes on the connection between polar decomposition and this problem:
Proof of the fact that $S = B(\theta)B(\theta)^\top$ has a solution if and only if $L_\star = L(\theta)$ has a solution.
If $L_\star = L(\theta)$ has a solution $\theta_\star$, then it follows that $$ B(\theta_\star)B(\theta_\star)^\top = L(\theta_\star)L(\theta_\star)^\top = L_\star L_\star^\top = S. $$
Conversely, suppose that $S = B(\theta)B(\theta)^\top$ has a solution $\theta_\star$. It follows that $L(\theta_\star)L(\theta_\star)^\top = S$. Now, we have $L(\theta_\star)L(\theta_\star)^\top = L_\star L_\star^\top$. From the above statements about the polar decomposition, it follows that $L_\star = L(\theta_\star)U$ for some orthogonal matrix $U$. That is, $$ L_\star I = L(\theta_\star)U $$ are $LQ$ decompositions of the same matrix. It follows that $L_\star = L(\theta_\star)$ and that $I = U$.
Note: I was wrong– it is not necessarily the case that a solution to $S = L_\star L_\star^T$ exists, but if such an $L_\star$ exists, then it is unique.