My question
Can we get any bound (depending on $A$ and $z$) of $\|A^+ z - z\|$, where $A^+$ is the Moore-Penrose inverse of the matrix $A\in \mathbb{R}^{n\times n}$ ?
Details
Actually this question came from a problem I have: Given $z^\star\in \mathbb{R}^n$, I must find $\underset{z\in \text{Im} A}{\text{argmin}} \|z-z^\star\|$ (or the min), and I thought that $A^+z^\star$ could be a good candidate, or at least a good approximation. Then I saw here (Two questions on the Moore–Penrose inverse) that the Moore-Penrose inverse "finds the lowest norm solution of $\|Ax-z^\star\|$". So it seems that this is the solution. But then can we have a bound on the error we make?
Could someone help ?
The Moore-Penrose inverse can be used to obtain least squares solution to the linear system $Ax \approx b$, in the sense $x_{\mathrm{LS}} := A^{\dagger} b$ is such that $x_{\mathrm{LS}} = \mathrm{argmin}_{x \in \mathbb{R}^{m}} \Vert Ax - b \Vert_2$. So the closest vector in $\mathrm{im}(A)$ to $b$ is actually $Ax_{\mathrm{LS}} = A A^{\dagger} b$. (The matrix $A A^{\dagger}$ is actually the orthogonal projection matrix onto the range of $A$; see Moore-Penrose inverse.)
In your case, you are trying to approximate $Az \approx z^{\star}$, so the closest vector in the range of $A$ to $z$ is actually $A A^{\dagger} z^{\star}$. (As mentioned in the comment, this only makes sense if $n = m$.) We have $\Vert A A^{\dagger} z^{\star} - z^{\star} \Vert_2 = \Vert (I - A A^{\dagger}) z^{\star} \Vert_2$. This is the norm of the component of $z^{\star}$ in the kernel of $A^T$ (the orthogonal complement of the range of $A$). If $A$ is full rank, then clearly this is zero since $z^{\star} \in \mathrm{im}(A)$. So suppose that $A$ is not full rank for non-triviality. Then we have the trivial bound $\Vert A A^{\dagger} z^{\star} - z^{\star} \Vert_2 \leq \Vert z^{\star} \Vert_2$, which is achieved if $z^{\star} \in \mathrm{ker}(A^T)$.
I'm not sure if we can give a more precise bound in terms of $A$ and $z$.