I faced an interesting problem when studying linear squares method. Let there be an overdetermined system $$Ax=b$$ with, say, $A=[30\times 8]$. By solving the normalized system $A^TAx=A^Tb$ we get an approximation $\hat x$.
Now I delete one out of 30 equations in the system, say the $i$th one. So, I get a new system $$A_{-i}x=b_{-i},$$ where $A_{-i}$ and $b_{-i}$ are obtained from $A$ and $b$ by deleting the $i$the row. Again, I solve the normalized system and get a completely different solution $\hat x_{-i}$. That is, the norms of $\hat x$ and $\hat x_{-i}$ are pretty close, but the values of elements of these vectors differ. For instance, we have $\frac{\|\hat x-\hat x_{-i}\|}{\|\hat x\|}=0.35$, which is quite a lot.
This seems rather counter-intuitive as we would expect that in an overdetermined system each particular equation does not contribute that much to the solution.
I tried to look into to the problem and checked singular values and conditioning numbers of both $A$ and $A_{-i}$. Nothing particular. The only difference appears in the second to last singular value. The cond. numbers are pretty much the same.
How can we treat such problems? In real problems we often add and delete some equations and expect that the solution will be the same. But it seems not to be the case. So, can we somehow estimate the contribution of a specific equation to the total solution? What is the right framework to analyse such phenomena?
UPDATE: To illustrate the problem I made a plot of $\frac{\|\hat x-\hat x_{-i}\|}{\|\hat x\|}$ for different $i$. It does not look like there are just a couple of outliers.
