Consider a linear system $Ax = b$
Assuming that $(A^\top A)^{-1}$ is invertible. The least square solution would be:
$ x = (A^\top A)^{-1}A^\top b $
I'm looking to form an expression for the uncertainty in the solution $x$ in the form of a covariance matrix. Assume that both $A$ and $b$ result from measurements with a given uncertainty. How can I can I get the covariance matrix for $x$?
Note: The general relationship I find given for the covariance matrix of the solution is:
$ \Sigma_{xx} = (A^\top A)^{-1}A^\top \Sigma_{bb} A (A^\top A)^{-1} $
but this does not account for uncertainties in $A$. How could I include their effect?
Use Total Least Squares (TLS) to get a solution of $Ax=b$ with uncertainties in $A,b$.
One example is you want to solve $Ax=b$. Assume there exist perturbations of $A$ and $b$ as $C$ and $d$ respectively.
Then, you want to solve $(A+C) x = b+d$ exactly for some suitably small choice of $C,d$. One way to do this is to find the perturbation $[C|d]$ that has the minimum Frobenius norm (provided it exists). It turns out by the Eckart-Young theorem, you can find $[C|d]$ by taking the singular value decomposition of $[A|b] = \sum_{i=1}^N \sigma_i u_i v_i^T$ where $\sigma_i$ are the singular values in decreasing order, and $u_i,v_i$ being the corresponding singular vectors and $[C|d]=-\sigma_N u_N v_N^T$.
Then, you just solve $(A+C) x = b+d$.
There are other variants on this idea (e.g. constrained TLS), and a fairly large literature overall.