EDIT: Sorry i messed up, I forgot a minus sign in front of the left hand side. I added it now.
I am not sure how to proceed with this. Given this non-linear ODE$$\partial_{t}u(t,x)=-\cot(t)\left[\frac{1}{8u(t,x)}\left(6u(t,x)^2-4A(x)u(t,x)+B(x)\right)\right]$$ fot $t\in(0,\pi)$, can I calculate the behavior of $u(t,x)$ for $t\to0,\pi$?
Treat the equation as an ODE in $t$ (which it is): the equation is first order and separable, so can be solved explicitly by integration. You obtain the equation $$ \sin(t) = f(u), $$ with $f(u)$ a function involving arctan and exp. This equality holds for all $t$ (and all $x$), so taking $t \to 0,\pi$ makes the left hand side zero. So, $u$ will converge to the zeroes of $f$ when $t \to 0,\pi$. So, all you have to do is find $f$, and determine its zeroes.