Least Squares understanding

Question

I am trying to understand the least squares method, I have found a simple enough example, but it doesn't fully explain the last part of it.

http://www.emathzone.com/tutorials/basic-statistics/example-method-of-least-squares.html

The part I am struggling with is this part

"Eliminate ‘a’ from equation (1) and (2), multiply equation (2) by 3 and subtract form equation (2), we get the values of ‘a’ and ‘b’."

First of all, why do you multiple by 3? Is this it just some arbitrary number that works in this least squares method. Secondly how do I go about this calculation. I have tried it, and I don't understand how I am meant to get 2 values. It does not show the working out, and I fail to see how you can get 2 values from a single subtraction. If I multiple equation 2 by 3, then subtract from equation 2, I would only get one value. How is 1.1 and 1.3 retrieved?

Math is not exactly my strongest point, but I need to understand it for task I am working on. Would someone be able to show me exactly how this solution was worked out

Thanks in advance

Answer 1

OK Adam, if you go through what the Least Squares actually does you end up needing to find two numbers $a$ and $b$ that satisfy the simultaneous equations

$$\begin{align}5a+15b&=25\\ 15a+55b&=88.\end{align}$$ Now there are many different ways of solving such a pair of equations (a linear system if you will). Here I will show you the way you might have solved them in school all those years ago.

First we need to understand what these equations are saying. When I write down

$$2x+3=11,$$

what we do first is we assume that $x$ solves the equation.

The second thing you need to realise is that if you have an equation, for example,

$$2x+3=11,$$ then you are allowed to add something to both sides... because when two numbers are equal, for example five and five, then adding three to both of them leaves you with two numbers that are still equal. So for example you could add $-3$ to both sides of $2x+3=11$ and you get

$$2x=8,$$

and these two numbers are still equal.

We are also able to multiply both sides by the same thing and we will end up with numbers that are still equal... in fact, if we are careful, we can do a lot of different things to the two numbers that are equal and they will remain equal. If we multiply both sides by a half here we end up with

$$x=4.$$

So what we do is we assume that $x$ solves the original equation and we are lead then to the conclusion that $x$ must equal four and indeed $2x+3=2(4)+3=8+3=11$.

The next thing we need to realise is that if you have two equations, then you can add these equations together. For example suppose I have $$\begin{align} 2x&=1 \\y&=0. \end{align}$$ What I can do is add $y$ to $2x$ and add zero to one. Why? Well because $y$ is equal to zero I am actually only adding the same thing to both sides.

Now if we look at our equations

$$\begin{align} 5a+15b&=25 \\ 15a+55b&=88, \end{align}$$ the problem is we have two numbers to find... i.e. we assume that $a$ and $b$ solve both equations and we want to go off and conclude that they must equal this or that...

The next problem is that, in general, one equation in two unknowns can have an infinite number of solutions. For example, if $x$ and $y$ are whole numbers, then

$$x+y=0,$$ has infinite number of solutions. You can have $x=1$, $y=-1$, $x=2$, $y=-2$, etc. Now the way you want to think is... if only I could find an equation in only one unknown. You are able to do that as follows.

Try and match up the number of $a$s (or $b$s), by multiplying both sides of the first equation by three:

$$3(5a+15b)=3(25).$$

Remember. $5a+15b$ is equal to 25 so if you multiply it by three you get 75. Now you need the little fact that $x(y+z)=xy+xz$... I assume and hope this is OK for you:

$$15a+45b=75$$.

Now we have that these are equal and allied to the second equation we know have

$$\begin{align} 15a+45b&=75 \\ 15a+55b&=88. \end{align}$$ Similarly to equation one, just to make things nice for us later, we can multiply both sides of the second equation by minus one to get the two pairs of equal numbers:

$$\begin{align} 15a+45b&=75 \\ -15a-55b&=-88. \end{align}$$

Now if we add the equations together --- i.e. add -88 to 75 and an equal thing ($-15a-55b$) to $15a+45b$ and then we get another pair of equal numbers (the $\Rightarrow$ stands for 'means that' or 'implies that'):

$$\begin{align} 15a+45b+(-15a-55b)&=75+(-88) \\ \Rightarrow -10b&=-13 \end{align}$$ Now multiply both sides by minus one:

$$10b=13,$$ and multiply both sides by $\frac{1}{10}$:

$$b=\frac{13}{10}=1.3.$$

Great! We know $b$. Now we know that the numbers $5a+15b$ and $25$ are equal from the very start... but $b$ is equal to 1.3 so that:

$$5a+15(1.3)=25.$$

I will leave the rest up to you.

Make sure to end by presenting your answer nicely with

$$Y(X)=a+bX.$$

Answer 2

In general, you end up with two equations, of the form

$$ C = na + rb \\ D = ra + sb $$

Here $n$ is the number of points, $r$ is the sum of the $x$-values, and $s$ is the sum of the squared $x$-values. In the example, $n = 5$, $r = 15$, and $s = 55$. I'm using letters rather than numbers so that you can see the general pattern.

I'm going to show how to perform algebraic operations on these two equations until we have formulas that express $a$ and $b$ in terms of $n$, $r$, and $s$; these formulas are the only ones you then need.

What I'm going to do is to multiply each equation by a constant, like changing $1 = 3 - 2$ into $4 = 12 - 8$ by multiplying everything by 4. Doing so doesn't change the solutions of the equations, as long as the constant I'm using isn't zero, but it can help make them easier to work with.

First, I'll multiply the top one by $s$:

$$ sC = sna + rs b \\ D = ra + sb $$

That's allowed, because the number $s$, which is the sum of the squares of a bunch of different numbers, will always be positive, so it's not zero.

Now I'm going to multiply the bottom equation by $r$, which might be zero. If it is, the new equation that I'll get will be just $0 = 0$, in disguise. So I'll keep both the new equation and the old one, to make sure I haven't lost any information:

\begin{align} sC &= sna + rs b \\ D &= ra + sb\\ rD &= r^2a + rsb \end{align}

Now I can subtract the bottom equation from the top one, to get this \begin{align} sC - rD &= (sna - r^2a) + rs b - rsb\\ D &= ra + sb. \end{align}

Notice how the two $rsa$ terms in the first equation cancel -- that's why I chose $s$ and $r$ as multipliers in the earlier steps. So now we have

\begin{align} sC - rD &= (sn - r^2)a\\ D &= ra + sb. \end{align}

From the first equation, we can solve for $a$ by dividing both sides by $r^2 - ns$. That's only valid if $r^2 - ns$ is nonzero. It's a slightly tricky fact about the least-squares set-up that this number will always be nonzero, as long as you have at least two distinct $x$-values. (If you had only one, it'd be tough to choose a single line to "best" fit through it, wouldn't it?) Anyhow, here's the result:

\begin{align} \frac{sC - rD}{sn - r^2} &= a\\ D &= ra + sb. \end{align}

Now that we have a formula for $a$, in the second equation, the values $D, r, a,$ and $s$ are all known, so we can find $b$. To do that in algebra, I'll bring the $ra$ to the other side:

\begin{align} \frac{sC - rD}{sn - r^2} &= a\\ D -ra &= sb. \end{align} and then divide through by $s$ (which is nonzero, so this is OK): \begin{align} \frac{sC - rD}{sn - r^2} &= a\\ \frac{D -ra}{s} &= b. \end{align}

Now we can substitute the answer we got for $a$ into the second equation to get a final result:

\begin{align} \frac{sC - rD}{sn - r^2} &= a\\ \frac{D -r\frac{sC - rD}{sn - r^2}}{s} &= b \end{align} which can be simplified a little bit (with some algebraic steps I'm not going to go through) to get

\begin{align} \frac{sC - rD}{sn - r^2} &= a\\ \frac{-rC +nD}{sn - r^2} &= b \end{align}

And that is the general solution for any least squares problem. Written out in terms of the things in the original article, you get

\begin{align} Q = n(\sum x_i^2) - (\sum x_i)^2 \\ \frac{(\sum x_i^2)(\sum y_i) - (\sum x_i)(\sum y_i^2)}{Q} &= a\\ \frac{-(\sum x_i)(\sum y_i) + n(\sum x_i y_i)}{Q} &= b \end{align} where I've defined the number $Q$ to make the second and third lines a little simpler.

Just to finish this off, let's look at the original problem, where $$ r = 15 \\ s = 55 \\ n = 5 \\ C = 25 \\ D = 88 $$

Then \begin{align} \frac{sC - rD}{sn - r^2} &= a\\ \frac{-rC +nD}{sn - r^2} &= b \end{align}

becomes

\begin{align} \frac{55 \cdot 25 - 15 \cdot 88}{55 \cdot 5 - 15^2} &= a\\ \frac{-15 \cdot 25 +5 \cdot 88}{55 \cdot 5 - 15^2} &= b \end{align} which simplifies to \begin{align} \frac{55}{50} &= a\\ \frac{65}{50} &= b \end{align} \begin{align} 1.1 &= a\\ 1.3 &= b \end{align}

Answer 3

The part I am struggling with is this part....

This part of the solution is just soving of system of linear equations but not LSM.