I have a diagram that looks like below:
I would like to find the variables $a$ and $b$ when the steady-state error becomes $0$ with unit acceleration as input $u(t) = \dfrac{t^2}{2}$.
I have found the following transfer function:
$$ L(s) = \dfrac{as^2 + bs + 1}{s(s+2)(s+8)} \\ G(s) = \dfrac{as^2 + bs + 1}{s^3 + 10s^2 + 16s + 1} $$
In order to locate $a$ and $b$, I have written the following equation:
$$ \begin{align*} E(s) = U(s) - Y(s) \\ Y(s) = P(s)E(s) \\ E(s) = \dfrac{1}{1 + P(s)}U(s) \end{align*} $$
$$ \begin{align*} e_a & = \lim_{s\to0}sE(s) = \lim_{s\to0}s\dfrac{1}{1+P(s)}R(s) \\ & = \lim_{s\to0}s*\dfrac{1}{1 + \dfrac{1}{s(s+2)(s+8)}}*\dfrac{1}{s^3} \end{align*} $$
However, the above equation will be ∞, and cannot be $0$ in this case. What is wrong with my procedure? Could it be the transfer function that I wrote is incorrect? Can anyone help me to figure out this? Thank you for your time and suggestion.
Based on the suggestion that Mr. Rollen S. has given, the steady-state error can be written as below:
$$ \begin{align*} e_a & = \lim_{s\to0}s\dfrac{s(s+2)(s+8) - (as^2+bs)}{s(s+2)(s+8)+1}*\dfrac{1}{s^3} \\ & = \lim_{s\to0}\dfrac{s^3+(10-a)^2+(16-b)s}{s(s+2)(s+8)+1}*\dfrac{1}{s^2} \\ & = \lim_{s\to0}\dfrac{s}{s(s+2)(s+8)+1} = 0 \end{align*} $$
with $a = 10, b = 16$.

The mistake is in the transfer function. First from the diagram we have that,
$$E(s) = U(s) - Y(s).$$
From the diagram we have that $Y(s) = P(s) \left( E(s) + C(s)\,U(s) \right)$ so,
$$E(s) = U(s) - P(s)\left( E(s) + C(s)\,U(s) \right).$$
Rewrite this as,
$$\left(1 + P(s)\right) E(s) = \left(1 - P(s)\,C(s)\right) U(s)$$
from which we deduce the transfer function from $U(s)$ to $E(s)$ is,
$$\frac{E(s)}{U(s)} = \frac{1 - P(s)\,C(s)}{1 + P(s)}.$$
Substituting your expressions for $P(s)$ and $C(s)$ yields,
$$\frac{E(s)}{U(s)} = \frac{s(s+2)(s+8) - (a s^2 + b s)}{s(s+2)(s+8) + 1}$$
If you simplify this, you can choose $a$ and $b$ so that this TF has a third-order zero at the origin which will cancel out with the poles of the given $u(t)$ which in turn achieves the objective of zero steady-state error.