Question:
Let the position of the particle be $\mathbf r(t) = (x(t),y(t))$. The equations of motion are
$$\ddot x = \frac{ay}{x^2+y^2} \qquad \ddot y = -\frac{ax}{x^2+y^2}$$
or equivalently,
$$\ddot {\mathbf r} = -a \nabla \tan^{-1}\bigg(\frac yx\bigg)$$
where $a>0$ is a constant.
The initial conditions are $\mathbf r(0) = (x_0,y_0)$ and $\dot{\mathbf r}(t) = \mathbf 0$, where $x_0,y_0>0$. Find the maximum speed attained by the particle in its subsequent motion.
Attempt:
So we want to find the maximum value of $\dot x^2 + \dot y^2$ (which is the squared velocity). Call this quantity $Q^2$ (of course, the actual quantity we are after is $Q$).
I started off by observing that
$$y\ddot x - x \ddot y = a$$
The left hand side can be factorised:
\begin{align} \implies & \frac{d}{dt}\big(y\dot x - x\dot y \big) = a \\ \implies & y\dot x - x\dot y = at + B \end{align}
for some constant $B$. Using initial conditions, we see that $B=0$, so
$$\implies y\dot x - x\dot y = at$$
Now differentiating $Q^2$ with respect to $t$:
$$\frac{dQ^2}{dt} = 2\dot x\ddot x + 2\dot y \ddot y = 2\dot x \frac{ay}{x^2+y^2} - 2\dot y \frac{ax}{x^2+y^2} = \frac{2a(y\dot x - x\dot y)}{x^2+y^2} = \frac{2a^2t}{x^2+y^2}$$
So it follows that $\frac{dQ^2}{dt}>0$ for all $t>0$. This implies that the speed is forever increasing - what have I done wrong? What is the correct approach?
I can't find anything wrong with your algebra, but you are worrying about something that is not an insurmountable problem.
Yes, the speed is forever increasing, but if $x^2+y^2$ is increasing faster than $t$ the square of the speed approaches some finite limit, asymptotically.
Try the effect of assuming that for large $t$, $$ x(t) = -v_x t + m_x \log t + k_x - \frac\alpha t + O(\frac1{t^2} )\\ y(t) = -v_y t + m_y \log t + k_y + \frac\beta t+O(\frac1{t^2} ) $$ and you will likely find that there are consistent values of $v_x, v_y$, which are then the asymptotic components of the velocity. Since the square of the velocity is monotonic increasing, you will not need to worry about oscillation when you say that the asymptotic value is the maximum value.