Suppose that $P(x)$ is a polynomial such that $$P(a+b)=6\big(P(a)+P(b)\big)+15a^2b^2(a+b)$$ for any complex numbers $a$ and $b$ satisfying $a^2+b^2=ab$, then find the polynomial $P(x)$.
I don't know how to even approach this one, it seems very complex. Your help is appreciated!
In this solution, $P$ is assumed to be an entire function (which is more generalised than just being a polynomial). Let $\omega_1,\omega_2$ be roots of $x^2-x+1=0$. We note that when $a=\omega_jt$ and $b=t$, we have $a^2+b^2=ab$, so that $$P\big((\omega_j+1)t\big)-6\big(P(\omega_j t)-6P(t)\big)-15\omega_j^2(\omega_j+1)t^5=0\tag{1}$$ for $j=1,2$.
Suppose that $$P(x)=\sum_{k=0}^\infty p_kx^k.$$ If $k\neq 5$, then the coefficient of $t^k$ of the LHS of $(1)$ is $p_k\cdot f_k(\omega_j)$, where $$f_k(x)=(x+1)^k-6(x^k+1).$$ If $p_k\ne 0$, then $f_k(\omega_j)=0$. For $k\geq 5$, we have $$\big|(\omega_j+1)^k\big|\geq |\omega_j+1|^k=\sqrt{3}^k=9\sqrt{3}$$ and $$\big|6(\omega_j^k+1)\big|\leq 6\big(|\omega_j|^k+1\big)=6\cdot 2=12.$$ Thus $$\big|(\omega_j+1)^k-6(\omega_j^k+1)\big|\geq \big|(\omega_j+1)^k\big|-\big|6(\omega_j^k+1)\big|\geq 9\sqrt{3}-12>0\,.$$ For $k=0,1,2,3,4$, we can easily verify by hand that $$(\omega_j+1)^k-6(\omega_j^k+1)\ne 0.$$ For $k=0,1$, this is trivial. For $k=2$, we note that $(x+1)^2-6(x^2+1)=-5x^2+2x-5$ is not divisible by $x^2-x+1$. For $k=3$, we note that $\omega_j^3=-1$ and $(\omega_j+1)^3\ne 0$. For $k=4$, note that $$(\omega_j+1)^4=\big((\omega_j+1)^2\big)^2=(3\omega_j)^2=9\omega_j^2$$ and $$\omega_j^4=-\omega_j,$$ so $$(\omega_j+1)^4-6(\omega_j^4+1)=9\omega_j^2+6\omega_j-6\ne0.$$ Thus $\omega_j$ is not a root of $f_k(x)$ for any integer $k\geq 0$. This is a contradiction, so $p_k=0$ for $k\ne 5$.
The coefficient of $t^5$ in $(1)$ is $$p_5(\omega_j+1)^5-6(\omega_j^5+1)-15\omega_j^2(\omega_j+1).\tag{2}$$ Noting that $$(x+1)^5-6(x^5+1)-15x^2(x+1)=-5(x+1)(x^2-x+1)^5,$$ we conclude that $$(\omega_j+1)^5-6(\omega_j^5+1)-15\omega_j^2(\omega_j+1)=0.\tag{3}$$ Subtract $(3)$ from $(2)$ to get $$(p_5-1)(\omega_j+1)^5=0.$$ Therefore $p_5=1$, so $P(x)=x^5$.