I managed to solve simple linear recursive relations with the help of generating functions, but I have trouble applying it to polynomial recurrent relations like this:
$$a_nx_n^2 - (2\sqrt{x_{n+1}} + a_nx_{n+1})x_n + x_{n+1}^{3/2}= 0$$
Where $x \ge 1 $.
By setting $x_n=y_n^2$,
$a_nx_n^2 - (2\sqrt{x_{n+1}} + a_nx_{n+1})x_n + x_{n+1}^{3/2}= 0 $ becomes $a_ny_n^4 - (2y_{n+1} + a_ny^2_{n+1})y^2_n + y_{n+1}^{3}= 0 $ which is polynomial but definitely not linear.
Note that $y_{n+1}$ is a quadratic in $y_n$ so it can explicitly be represented in terms of $y_n$ using the quadratic formula.