I did the first part correctly, but when I computed $B=(X^T X)^{-1} X^T Y$, I got the wrong answer. I double checked my calculations on my calculator, all was fine. I think it's because of the unknown parameter vector, but I dont know how to solve this then? Isn't that too many unknowns?
You have data points $(x_i, y_i), 1\le i \le 5$. Your function is of the form: $$ y = \beta_1 x + \beta_2 x^2 $$ Considering your data you can rearrange in vectors and matrices to get: $$ \mathbf{y} = \mathbf{X}\boldsymbol{\beta} $$ where $\mathbf{y} = \begin{bmatrix} y_1 \\ \vdots \\ y_5 \end{bmatrix}$ , $\boldsymbol{\beta} = \begin{bmatrix}\beta_1 \\ \beta_2\end{bmatrix}$, and $ \mathbf{X}= \begin{bmatrix} x_1 & x_1^2 \\ \vdots\\ x_5 & x_5^2 \end{bmatrix} $. You can get your unknown vector of parameters using $$ \mathbf{y} = \mathbf{X}\boldsymbol{\beta} \Rightarrow \mathbf{X}^T\mathbf{y} = \mathbf{X}^T\mathbf{X}\boldsymbol{\beta} \Rightarrow \boldsymbol{\beta} = (\mathbf{X}^T\mathbf{X}) ^{-1}\mathbf{X}^T\mathbf{y} $$ This corresponds to the least-squares solution and the matrix $(\mathbf{X}^T\mathbf{X}) ^{-1}\mathbf{X}^T$ is the (left) Moore-Penrose (pseudo) inverse of $\mathbf{X}$.
Doing the calculation I found (rounded to two decimals) $\boldsymbol{\beta} = \begin{bmatrix} 1.76\\ -0.20 \end{bmatrix}$. Your function (allowing you to generate the curve) can then be written as: $$ y = 1.76 x - 0.20 x^2. $$
Here is a graph showing your data points and the curve drawn using the function above.