For the operator defined as polynomial is the boson creation and annihilation operators $\hat{a}$, $\hat{a}^\dagger$ such that $[\hat{a},\hat{a}^\dagger] = 1$
$$\hat{L} = A\hat{a}^2 + B\hat{a}^{\dagger2} + C\hat{a}^\dagger\hat{a} + D\hat{a}^\dagger + E\hat{a} + F,$$
How can I introduce the Bogoliubov transformation $\hat{b} = \mu\hat{a} +\nu\hat{a}^\dagger$, such that the operator $\hat{L}$ can be written diagonal in $\hat{b}$?
It is not quite clear to me what diagonal means for you, I'll assume you mean this $$ \hat{L} = G \hat{b}^\dagger \hat{b} + H\hat{b}+ J\hat{b}^\dagger + K. $$ Now we can just expand and compare coefficients. We have $$ \hat{b}^\dagger \hat{b} = (\mu \hat{a} + \nu \hat{a}^\dagger) (\mu \hat{a} + \nu \hat{a}^\dagger) = \mu^2 \hat{a}^2 + \nu^2 {a}^{\dagger 2} + \mu \nu \hat{a}^\dagger \hat{a} + \mu \nu \hat{a} \hat{a}^\dagger. $$ Now we use the anticommutation rule and get $$ \hat{b}^\dagger \hat{b} = \mu^2 \hat{a}^2 + \nu^2 {a}^{\dagger 2} + 2\mu \nu \hat{a}^\dagger \hat{a} + \mu \nu. $$ Hence, we have $$ G \hat{b}^\dagger \hat{b} + H\hat{b} + J\hat{b}^\dagger + K = G\mu^2 \hat{a}^2 + G\nu^2 {a}^{\dagger 2} + 2G\mu \nu \hat{a}^\dagger \hat{a} + G\mu \nu + H\mu \hat{a} + H\nu \hat{a}^\dagger + J \nu \hat{a} + J\mu \hat{a}^\dagger + K. $$ Collecting terms gives $$ G \hat{b}^\dagger \hat{b} + H\hat{b} + J\hat{b}^\dagger + K = G \mu^2 \hat{a}^2 + G \nu^2 \hat{a}^\dagger + 2 G \nu \mu \hat{a}^\dagger \hat{a} + (H\mu + J \nu) \hat{a} + (H\nu + J \mu) \hat{a}^\dagger+ (G\mu \nu + K). $$ If we want $$ G \hat{b}^\dagger \hat{b} + H\hat{b} + J\hat{b}^\dagger + K = \hat{L} = A \hat{a}^2 + B \hat{a}^{\dagger 2} + C \hat{a}^\dagger \hat{a} + D \hat{a}^\dagger + E\hat{a} + F, $$ then we have to match the coefficients, i.e. $$ \begin{cases} A &= G \mu^2,\\ B &= G \nu^2, \\ C &= 2G\mu \nu,\\ D &= H\mu + J \nu,\\ E &= H\nu + J \mu,\\ F &= G\nu \mu + K. \end{cases} $$ This gives you also some restrictions on when you can write it in diagonal form.