Linear Algebra: Residual $A x - b$

Question

I'm doing a course on Applied Linear Algebra. Which, for the most part, is about applications in Octave, so there isn't much about the happenings behind a lot of the mechanics of everything.

I'm faced with the statement: "You can use the condition number to estimate the accuracy at which Octave solves for $x$ in $A x = b$. "First, we define the residual which is $A x - b$."

And I'm battling to understand this residual "$A x - b$". I'm trying to find some answers as to what this residual is and what to do with it.

Any help or reading material I can explore will be greatly appreciated.

Answer 1

By numerical solution of $Ax=b$ we obtain $\bar x$.

If $\bar x$ was an exact solution it should be $$A\bar x=b \implies A\bar x-b=0$$

since we always have some kind of numerical error or approximation we have a residual (vector) that is

$$r=A \bar x-b\neq 0$$

We can extimate the error by the norm |r|.

Answer 2

You can consider a matrix $A \in \mathbb{R}^{m{\times}n}$, as a procedure to move point (or vector) x in $\mathbb{R}^n$ to a point b in $\mathbb{R}^m$, denoted $Ax=b$. Usually, $m=n$, meaning the matrix is square.

An example would be to have a matrix $A \in \mathbb{R}^{3{\times}3}$ and x in $\mathbb{R}^3$, which can be considered our space. The center of the sun could be defined as the origin i.e. $(0, 0, 0)$. If we compute $Ax=b$ , we compute how the matrix $A$ moves point $x$ in space to $b$. The trajectory of Heavy Falcon can then be described as $x_{i+1}=Ax_{i}$ and $x_{0}$ would then be Cape Canaveral.

A linear system of equation $Ax=b$ can be reformulated as: given the "moving procedure" $A$ and my final position $b$, where did I start?

The residual is the difference between $Ax_{guess}$ and $b$, meaning we guess that we started from $x_{guess}$ and since we know the "moving procedure" $A$, we end up at $Ax_{guess}$. If your $x_{guess}$ is correct, we end up at $b$. Usually the distance (euclidean norm) is used: $||Ax_{guess}-b||_2$.