Let us say that we modify the law of gravitation to the following force field
$$ \vec{F}(r) = \frac{-GMm}{r^2}\hat{r} + \alpha\frac{|L|^2}{m^2 r^3}\hat{r} $$
where,
$L = m_2(\vec{r} \times \dot{\vec{r}} )$ which is the angular momentum of $m$ in the reference frame of $M$.
$M$ is mass of parent body which is stationery in our reference frame.
$m$ is mass of the moving body.
$\vec{r}$ is position vector of $m$ with $M$ as the reference (origin).
$\dot{\vec{r}}$ is the velocity of $m$ with M being the reference or origin again.
First Question
Find the equation of orbit of a planet around the Sun and draw an approximate orbit for $\alpha \ll 1.$
Second Question
For the Sun-Mercury system find the value of $\alpha$. Here $M$ is the Sun.
I could not solve this question. However I feel that Newton's Theorem of revolving Orbits is somehow needed for the solution. I would like to see a proof of the theorem as well as a complete solution for the problem.
I did not post this question for Physics SE mainly because (i) this question mainly needs knowledge of vector calculus (or so I feel) and (ii) My experience is that Math SE is a much better community so I prefer posting a borderline Maths-Physics question on Mathematics SE rather than Physics SE.
Edit 1
I know how to derive the two body problem by using the Rung Lenz vector method. So any answers may skip over details such as how to prove $\vec{L}$ is constant etc.
Edit 2
An article by Baez on the inverse cube force law.
Let's start by writing the kinetic energy for the planet (strictly speaking you also need to consider the motion of the sun, but for mercury $M_{\rm sun} \gg M_{\rm planet}$ so I will ignore the motion of the center of mass)
$$ T(r,\theta,\phi) = \frac{1}{2}m(\dot{r} + r^2\dot{\theta}^2 + r^2\sin^2\theta\dot{\phi}^2) \tag{1} $$
And the potential energy is just
$$ V(r) = -G\frac{Mm}{r^2}+ \alpha\frac{l^2}{2m^2 r^2} \tag{2} $$
Note that ${\bf F} = -\nabla V$, that's a good check. The Lagrangian can then be written as (down below you'll see this is completely equivalent to writing Newton's equations)
$$ L = T - V = \frac{1}{2}m(\dot{r}^2 + r^2\dot{\theta}^2 + r^2\sin^2\theta\dot{\phi}^2) + G\frac{Mm}{r^2} - \alpha\frac{l^2}{2m^2 r^2} \tag{3} $$
Now we just need to derive the equations of motion. In general if $q$ is a generalized coordinate ($q = \{r,\theta,\phi\}$) then
$$ \frac{{\rm d}}{{\rm d}t}\left(\frac{\partial L}{\partial \dot{q}}\right) - \frac{\partial L}{\partial q} = 0 \tag{4} $$
which is known as the Euler-Lagrange equation $^\color{red}{1}$.
$$ \frac{{\rm d}}{{\rm d}t}\left(\frac{\partial L}{ \partial \dot{\phi}}\right) - \frac{\partial L}{\partial \phi} = \frac{{\rm d}}{{\rm dt}}(mr^2\sin^2\theta \dot{\phi}) - (0) = 0 $$
From this you conclude that the term in parenthesis is a constant, which is nothing more than the angular momentum $l$. I will not go into the details (this is getting pretty long) but this ensures that the motion will be on a plane: $\theta = \pi/2$, we have then
$$ \frac{{\rm d}}{{\rm d}t}(mr^2\dot{\phi}) = 0 ~~~\Rightarrow~~~ l = mr^2\dot{\phi} \tag{5} $$
For this case we get (remember to consider the fact that $l$ is a constant)
$$ \frac{{\rm d}}{{\rm d}t}(m\dot{r})-\left(-G\frac{M m}{r^2} + \alpha\frac{l^2}{m^2r^3}+ mr \dot{\phi}^2\right) = 0 $$
Which reduces to
$$ m\frac{{\rm d}}{{\rm d}t}\dot{r} = - G\frac{mM}{r^2} + \frac{l^2}{mr^3}\left(1 - \frac{\alpha}{m} \right) \tag{6} $$
You can gain some insight by realizing this can be written as
$$ m\frac{{\rm d}}{{\rm d}t} \dot{r} = -\frac{{\rm d}}{{\rm d}r}V_{\rm eff}(r) $$
for some potential $V_{\rm eff}$ (please derive it) and finding the stable points. The variable $r$ will then more around such point
I will leave the remaining details for you to complete. Good luck!
$^\color{red}{1}$ You can convince yourself that this reduces to Newton's equations with a simple example, in 1D $T = m\dot{x}^2/2$ so that $L_{1D} = m\dot{x}^2/2-V$ and Eq. (4) becomes
$$ \frac{{\rm d}}{{\rm d}t}(m \dot{x}) - (-V'(x)) = 0 ~~~\Rightarrow~~~ m\ddot{x} = -V'(x) = F(x) $$