The given is $$r(t)=\frac{3}{2}t^2\pmb i-4t\pmb j+\frac{1}{2}t^3\pmb k$$ Taking the first two derivatives: $$r^{'}(t)=3t\pmb i-4\pmb j+\frac{3}{2}t^2\pmb k$$ $$r^{''}(t)= 3\pmb i+3t\pmb k$$ Taking the dot product of $r^{'}(t) \bullet r^{''}(t)$ $$9t+\frac{9}{2}t^2$$ And now the cross product of $r^{'}(t) \mathsf X r^{''}(t)$ $$-12t\pmb i+\frac{9}{2}t^2\pmb j+12\pmb k$$
I know I did the dot product correct but the website we are using says its wrong, I then did the cross product but it also says that is wrong as well.
You should check your calculations. For the first thing, $r'(t)\cdot r''(t)=9t+\frac{9}{2}t^3.$ Indeed, it looks to me like you made a mistake in multiplying the $k$ components together.
As for the cross product: it looks like you made the common mistake of not putting a "-" before the $\vec{j}$ term. It becomes: $$ r'(t)\times r''(t)=-12t\vec{i}-\frac{9t^2}{2}\vec{j}+12 \vec{k}.$$