So I am curious what/if anything the euclidean distance between two matrices means? I have an algorithm that computes the solution of this matrix $B$ iteratively and one of the questions it asks is if $| B^{i+1} - B^{i} |_2 < \epsilon$ then a First Order, Second Order Necessary condition has been met or even a Second Order Sufficient Condition.
I would appreciate some intuitive insight if possible.
Thanks
Informally, you can think of the condition $\|B^{i+1} - B^i\|_2 \le \epsilon$ as "$B$ have not changed much in the last iteration". When $\|B^{i+1} -B^i\|_2 = 0$, i.e. $B^{i+1} = B^i$, your method have converged (and with suitable assumptions, you reached optimum). $\|B^{i+1} - B^i\|_2 \le \epsilon$ is "close enough" for convergance in practice (when $\epsilon$ is small enough).
For example, assume you apply a first-order gradient descent method on a problem $\min_B f(B)$, where $f$ is convex and differentiable. Specifically, assume that you run $B^{i+1} = B^{i} - \eta \nabla f(B^i)$ with constant step size $\eta$ until the condition $\|B^{i+1} - B^i\|_2 \le \epsilon$ is met. In this scenario, $\|B^{i+1} - B^i\|_2 = \eta \|\nabla f(B^i)\|_2$, hence the condition is $\|\nabla f(B^i)\|_2 \le \frac{\epsilon}{\eta}$. Since $f$ is convex and differentiable, $\|\nabla f(B^i)\|_2 = 0$ is a sufficient condition for optimality. Demanding $\|B^{i+1} - B^i\|_2 \le \epsilon$ is now equivalent to "the gradient is very small now, so we are close enough to the minimum".