For such polynomial $P(x)$,
$$P(x^4) = ax^8+(b+1)x^5+(ab)x^4+(a-1)x^2 + 2b-2$$
Find the remainder of the division of $P(x^3-3)$ with $x+1$.
We might have to find $P(x)$ first. My attempt is shown below
$$P(x^4) = a(x^4)^2 + (b+1)(x^4)x+(ab)x^4 + (a-1)\dfrac{x^4}{x^2}+2b-2$$
So
$$P(x) = ax^2 +(b+1)x^2 + (ab)x + \color{red}{(a-1)x^{-1}}+2b-2$$
However, the degree of a polynomial should be a natural number. Correspondingly I've gone wrong somewhere.
Regards
From $$P(t) = at^2+(b+1)t^{5/4}+(ab)t+(a-1)t^{1/2} + 2b-2$$ you deduce that $b=-1$ and $a=1$ for $P$ to indeed be a polynomial.
Then the requested remainder is
$$P((-1)^3-3)=(-4)^2-(-4)-4=16.$$