The points are $ (2,1.5),(−1,1.7),(1,1.9)$. I did:
$A = \left [ \begin{matrix} 1 & 2 & 4 \\ 1 & -1 & 1 \\ 1 & 1 & 1 \end{matrix} \right ] \\ x = \left [ \begin{matrix} b \\ 3 \\ a \end{matrix} \right ]\\ b = \left [ \begin{matrix} 1.5 \\ 1.7 \\ 1.9 \end{matrix} \right ]$
I solved for $$A^TAx = A^Tb$$
I got the Cholesky decomposition of $A^TA$
Then I solved $Ly = A^Tb$ and got $$ \left [ \begin{matrix} 51/10 \\ 16/5 \\ 48/5 \end{matrix} \right ]$$$
The next part is where I am having a problem. Now I have to solve $L^tx = y$. I already have some values for x, so I multiplied $L^tx$ and made it equal to y and got a system of equations
How do I solve this?



$3$ is not an unknown variable, so your solution is incorrect. We have $$ \left\{\begin{array}{rrrrr} 2^2\cdot a &+3\cdot 2&+b&= &1.5 \\ (-1)^2\cdot a&+3\cdot(-1)&+b&= &1.7 \\ 1^2\cdot a&+3\cdot 1&+b&= &1.9 \\ \end{array}\right. $$ or $$ \left\{\begin{array}{rrrrr} 4a&+&b&= &-4.5 \\ a&+&b&= &4.7 \\ a&+&b&= &-1.1 \\ \end{array}\right. $$ or $$ Ay=d, $$ where $$ A=\left(\begin{array}{ll} 4 & 1\\ 1 & 1\\ 1 & 1\\ \end{array}\right),\quad d=\left(\begin{array}{r} -4.5 \\ 4.7 \\ 1.9 \\ \end{array}\right),\quad y=\left(\begin{array}{r} a \\ b \\ \end{array}\right). $$ This system has no solutions, thus, there is no curve of the form $ax^2+3x+b$ which goes exactly through the given points. It is only possible to find a least squares solution. This solution won't make a lot of practical sense since $a+b$ can't be equal to $4.7$ and $-1.1$ at the same time even approximately. $$ y_{ls}=A^{+}d=(A^TA)^{-1}A^Td=\left(\begin{array}{r} -2.1 \\ 3.9 \\ \end{array}\right). $$ Hence, the answer is $-2.1x^2+3x+3.9$.