I need the transfer function of:
$$K_1 \dot Q(t) + K_2 Q(t) + K_3 Q(t)^2 + K_4 = 0 $$
Where $K_1$, $K_2$, $K_3$ and $K_4$ are constants that depend on the system parameters.
For this I need to apply the Laplace transform, but I don't know how to get the $\mathscr{L}\{Q^2(t)\}$. Any tips?
First of all transfer functions can only be defined for linear time invariant systems and your system is nonlinear. However, it is still possible to find an expression for the Laplace transform of your expression using the convolution theorem. Namely, the property of convolution in the time domain being equivalent to multiplication in the frequency domain also works the other way around, so multiplication in the time domain is equivalent to convolution in the frequency domain which gives $\mathcal{L}\{Q^2(t)\}(s)=(\mathcal{L}\{Q(t)\}*\mathcal{L}\{Q(t)\})(s)$ with $*$ denoting convolution.