Error propagation in a simple linear model (asked by a non math-major researcher).

Question

I have the following linear problem in a matrix form $Ax=Y$, where $A$ is a coefficient matrix, $x$ is a vector of unknown parameters and $Y$ is a vector of observed data. Matrix $A$ is usually over-determined and often rank deficient; it has 1000s of rows and columns. I find a solution by applying the pseudo-inverse (SVD): $x=A^{-1}Y$. I compute a conditioning number (equal about 5) to learn how the input errors propagate to the output errors. Can I determine by looking at matrix $A$ if each unknown parameter in vector $x$ is resolved well and if there is an unwanted correlation between some $x_{i}$ and $x_{j}$. I have done tests using some synthetic data and they are good, but here I would like to know if anything can be determined from the matrix $A$ alone.

Answer 1

The covariance has the following transformation property \begin{equation} Cov[A\vec{x}] = A Cov[\vec{x}]A^{\intercal}, \end{equation} where $Cov[.]$ is a covariance matrix. This property can be proved as follows \begin{equation} Cov[A\vec{x}] = E[(A\vec{x} - E[A\vec{x}])(A\vec{x} - E[A\vec{x}])] \\ = E[A(\vec{x} - E[\vec{x}])(\vec{x} - E[\vec{x}])A^{\intercal}] \\ = A E[(\vec{x} - E[\vec{x}])(\vec{x} - E[\vec{x}])]A^{\intercal} \\ = A Cov[\vec{x}]A^{\intercal}, \end{equation} where $E[.]$ is the expectation. I.e. in your case \begin{equation} A Cov[\vec{x}]A^{\intercal} = Cov[\vec{y}], \end{equation} so the covariance matrix for $\vec{x}$ has the following form \begin{equation} Cov[\vec{x}] = A^{+}Cov[\vec{y}](A^{\intercal})^{+}, \end{equation} where $A^{+}$ is pseudoinverse of $A$ Sometimes even better to write the answer in terms of inverse covariance matrices: \begin{equation} Cov^{-1}[\vec{x}] = A^{\intercal}Cov^{-1}[\vec{y}]A. \end{equation}

There is also well-known inequality that links relative perturbations of the solution $\vec{x}$ and relative perturbations of the right hand side $\vec{y}$: \begin{equation} \frac{\|\Delta\vec{x}\|}{\|\vec{x}\|} \leq \frac{cond(A)}{1 - cond(A)\frac{\|\Delta A\|}{\|A\|}}\left(\frac{\|\Delta\vec{y}\|}{\|\vec{y}\|} + \frac{\|\Delta A\|}{\|A\|}\right), \end{equation} where $\Delta\vec{y}$ is the perturbation of the vector $\vec{y}$, $\Delta\vec{x}$ is the perturbation of $\vec{x}$ and $\Delta A$ is the perturbation of the matrix $A$. If the matrix perturbation is zero, the inequality has a simpler form \begin{equation} \frac{\|\Delta\vec{x}\|}{\|\vec{x}\|} \leq cond(A)\frac{\|\Delta\vec{y}\|}{\|\vec{y}\|}. \end{equation}

The last two inequalities can be simply obtained using the definition of the condition number $cond(A)=\|A^{+}\|\|A\|$.