Problem: Let $V = \mathbb{R}[X]_{\leq 4}$ be the vectorspace of all polynomials of degree at most $n$, and let $\alpha = \left\{1, 1 +x, (1+x)^2, (1+x)^3, (1+x)^4\right\}$ be a basis for $V$. Find the coordinates of the vector $x+x^3 + x^4$ with respect to this basis.
Attempt at solution: We are looking for scalars $\lambda_i$ such that \begin{align*} x+ x^3 + x^4 = \lambda_1 (1) + \lambda_2 (1+x) + \lambda_3 (1+2x+x^2) + \lambda_4 (1+3x + 3x^2 + x^3) + \lambda_5 (1+4x+6x^2 +4x^3 + x^4) \end{align*} where I expanded the basis vectors. Checking for corresponding coefficients, this gives rise to the following system of equations: \begin{align*} \begin{cases} \lambda_1 + \lambda_2 + \lambda_3 + \lambda_4 + \lambda_5 &= 0 \\ \lambda_2 + 2 \lambda_3 + 3\lambda_4 + 4 \lambda_5 &= 1 \\ \lambda_3 + 3 \lambda_4 + 6 \lambda_5 &= 0 \\ \lambda_4 + 4 \lambda_5 &= 1 \\ \lambda_5 &= 1 \end{cases} \end{align*} I put this in an augmented matrix and row reduced it: \begin{align*} & \left(\begin{array}{ccccc|c} 1 & 1 & 1 & 1 & 1 & 0 \\ 0 & 1 & 2 & 3 & 4 & 1 \\ 0 & 0 & 1 & 3 & 6 & 0 \\ 0 & 0 & 0 & 1 & 4 & 1 \\ 0 & 0 & 0 & 0 & 1 & 1 \end{array}\right) \begin{matrix} \xrightarrow{R_4 \rightarrow R_4 - 4 R_5} \\ \xrightarrow{R_3 \rightarrow R_3 - 6 R_5} \\ \xrightarrow{R_2 \rightarrow R_2 - 4R_5} \\ \xrightarrow{R_1 \rightarrow R_1 - R_5} \end{matrix} \left(\begin{array}{ccccc|c} 1 & 1 & 1 & 1 & 0 & -1 \\ 0 & 1 & 2 & 3 & 0 & -3 \\ 0 & 0 & 1 & 3 & 0 & -6 \\ 0 & 0 & 0 & 1 & 0 &-3 \\ 0 & 0 & 0 & 0 & 1 & 1 \end{array}\right) \\ \begin{matrix} \xrightarrow{R_3 \rightarrow R_3 - 3R_4} \\ \xrightarrow{R_2 \rightarrow R_2 - 3 R_4} \\ \xrightarrow{R_1 \rightarrow R_1 - R_4} \end{matrix} & \left(\begin{array}{ccccc|c} 1 & 1 & 1 & 0 & 0 & -4 \\ 0 & 1 & 2 & 0 & 0 & 6 \\ 0 & 0 & 1 & 0 & 0 & 3 \\ 0 & 0 & 0 & 1 & 0 & -3 \\ 0 & 0 & 0 & 0 & 1 & 1 \end{array}\right) \end{align*} Then with backward substitution I found that \begin{align*} \begin{cases} \lambda_5 &= 1 \\ \lambda_4 &= -3 \\ \lambda_3 &= 3 \\ \lambda_2 &= 0 \\ \lambda_1 &= -7 \end{cases} \end{align*} But when I plug this in the equation above, it doesn't give me the right vector. Can someone tell me where I went wrong please?
In the last row reduction when you do $R_1\leftarrow R_1-R_4$ the last number in $R_1$ should be $-1-(-3)=-1+3=2$, but you wrote $-4$. This makes the very last back-substitution, for $\lambda_1$, wrong.
I don't understand what you are doing. Your original matrix was upper-triangular, so you could have done back-substitution from there, so no row reductions at all were needed. And you stopped before you got to a reduced row echelon form, where no back-substitution would have been needed. What was your point of doing a few row reductions without going all the way to reduced row echelon form?