I'm trying to solve a linear system such as $Ax = b$. This is related to a measurement system I'm working on and the nature of the equations is such that $A$ is singular in the absence of errors.
When I use made up perfect data with no errors, I can solve the system using an SVD and get the exact correct answer. However, when I use real measurement data, which has a good amount of error in it, the solution is further off then it should be.
My understanding is that because my matrix is highly singular errors are magnified. Is there any way I can improve this or is this the best I can expect given the circumstances?
EDIT: The matrix is always larger than a $4 \times 4$ and its rank is always 4 (if anyone cares).
EDIT2: Context - The problem is related to a GPS system but I'm not using the conventional GPS solutions. $A$ is $n \times n$ with n being the number of satellites in view. The problem is fully defined with $n = 4$ since there are four unknowns. It becomes overdefined for more than four satellites. Because of the errors in the signals from each satellite, a choice of only four gives a poor estimate, so the goal is to solve the overdefined problem to achieve better accuracy. That is where I'm running into this issue.
One way to attack the problem is "incremental'. I mean by that
1) Add to the first four constraints a fifth constraint chosen among the remaining ones. Thus, for the resulting new $5 \times 5$ matrix, obtain its singular values and compute a measure $m$ of perturbation of the set of the first four ones, for example
$$m=\sum_{k=1}^4 |\sigma_k/s - \sigma'_k/s'| \ \ \ \ \text{or} \ \ \ \ m=\sum_{k=1}^4 (\sigma_k/s - \sigma'_k/s')^2$$
where $\sigma_k$ is for the $4 \times 4$ matrix, and $\sigma'_k$ is for the $5 \times 5$ matrix ($\sigma'_5$ is not considered), and $s$ (resp $s'$) is the sum of the $\sigma_k$ (resp. $\sigma'_k$ ).
2) Do that for all remaining constraints
3) Keep the constraint (i.e., its matrix) that has the smaller $m$ (i.e., is the more coherent with the previous measures).
4) Then repeat (do the same thing by adding a sixth constraint with the augmented 4+1 system of constraints)...