Can we choose $a_{0} \in \mathbb{R}$ so that the sequence $(a_{j})_{j=0}^{\infty}$ defined via $a_{j+1} = a_{j}^2\sin(a_{j})$ is bounded but not convergent?
Note that if $p$ is a fixed point of the function $f(x) = x^2\sin(x)$ then $p, f(p), f(f(p)),...$ converges to $p$. If $p < 1$ then the sequence mentioned in the previous sentence converges to 0. I don't know whether we can choose $a_{0} \in \mathbb{R}$ so that the above sequence is divergent.
Edit: it seems achille hui has found an $a_{0}$ which is the root of $x\sin (x) = -1$, this $a_{0}$ satisfies the first question. Can we now choose an $a_{0}$ so that the above sequence is unbounded (divergent)?
Yes it can be done, albeit in a peculiar way.
Say $x_1\in[2\pi,{5\pi\over2}]$. We do not choose a specific value as of now; we just say that it is somewhere there in that interval. This puts $x_2=f(x_1)$ someplace between $0$ and ${25\pi^2\over4}\approx61.68$, which is more than enough for our purposes. Let's say $x_2\in[4\pi,4\pi+{\pi\over2}]$. This, in retrospect, narrows down the position of $x_1$ to a smaller interval. Also, this gives us ample room for $x_3$, and we may choose a suitable segment for it, say, $x_3\in[6\pi,6\pi+{\pi\over2}]$, which in turn will give us even more precise positions of $x_2$ and $x_1$.
Continue ad infinitum. The nested intervals defining $x_1$ will converge to a point, and since we chose every interval so that $x_n\geqslant2\pi n$, we may be sure that the sequence $(x_n)$ is unbounded and divergent.
(The drawing is not to scale.)
So it goes.