We know that, for some $H\in\mathbb{C}[x]$,
$$P = QH+R$$
Hence,
$$P-R=QH$$
Given that $Q$'s roots are $\{i,-i\}$, the Remainder Theorem for polynomial rings guarantees that evaluating $(P-R)(\pm i)=0$. So we have the following system of equations
$$\begin{cases} 3i^{120}+\sqrt{2}i^{19}+ai^{11}+\sqrt{2}i^9+3i^2+5i+b+2 =0 \\ 3(-i)^{120}+\sqrt{2}(-i)^{19}+a(-i)^{11}+\sqrt{2}(-i)^9+3(-i)^2+5(-i)+b+2 =0 \end{cases}$$
which entails that
$$a=-5-\sqrt{2},b=2$$
Is this more or less correct and properly justified?
Right method, but there are sign errors: $$\begin{cases} 3i^{120}+\sqrt{2}i^{19}+ai^{11}+\sqrt{2}i^9+3i^2\color{red}-5i+b\color{red}-2 =0 \\ 3(-i)^{120}+\sqrt{2}(-i)^{19}+a(-i)^{11}+\sqrt{2}(-i)^9+3(-i)^2\color{red}-5(-i)+b\color{red}-2 =0 \end{cases} \Rightarrow \\ \begin{cases} \require{cancel}\cancel{3}-\cancel{\sqrt{2}i}-ai+\cancel{\sqrt{2}i}-\cancel{3}\color{red}-5i+b\color{red}-2 =0 \\ \cancel3+\cancel{\sqrt{2}i}+ai-\cancel{\sqrt{2}i}-\cancel{3}+5i+b\color{red}-2 =0 \end{cases} \Rightarrow a=-5,b=2.$$ Alternatively, note: $$3x^{120}+3x^2\equiv 0\pmod{x^2+1}\\ \sqrt{2}x^{19}+\sqrt{2}x^9 \equiv 0\pmod{x^2+1}\\ ax^{11}+b\equiv 5x+2\pmod{x^2+1} \Rightarrow \\ ax^{11}+b=g(x)(x^2+1)+5x+2 \Rightarrow \\ ai^{11}+b=g(i)\cdot 0+5i+2\Rightarrow \\ -ai+b=5i+2 \Rightarrow \\ a=-5,b=2.$$