The Question
Consider the extremal curves in the plane which make the line integral $$I=\int_P^Q \left(y^2dx+\frac{dy}{dx}dy\right)$$ stationary, where $P=(0,2)$ and $Q=(1,b)\ (b>0)$. To do so, first rewrite the integral in the standard form of the fundamental problem of variational calculus, involving $y(x)$. Then obtain the form of the curve(s).
Problem
I have been given the above question, I am posting on here to see if anyone can offer some advice as to where/ how to start with this problem, as I am clueless, Thanks in advance.
You can write the problem as
$$ I = \int_{P}^Q \left(y^2\frac{dx}{dt}dt + \frac{dy/dt}{dx/dt} \frac{dy}{dt}dt \right) = \int_P^Q \left(y^2x' + \frac{y'^2}{x'}\right)dt = \int_P^Q L(x,y,x',y')dt $$
Where $L$ is given by
$$ L = y^2x' + \frac{y'^2}{x'} $$
To solve this problem you can use the Euler-Lagrange equations:
\begin{eqnarray} \frac{d}{dt}\left(\frac{\partial L}{\partial y'}\right) - \frac{\partial L}{\partial y} &=& 0 \\ \frac{d}{dt}\left(\frac{\partial L}{\partial x'}\right) - \frac{\partial L}{\partial x} &=& 0 \\ \end{eqnarray}
For example, the first equation becomes
$$ \frac{d}{dt}\left(2\frac{y'}{x'}\right) - (2yx') = 0 $$
Rearranging we arrive to
$$ y(t) x'(t)^2+\frac{x''(t) y'(t)}{x'(t)}=y''(t) \tag{1} $$
Similarly for the $x$:
$$ x''(t) y'(t)-x'(t) y''(t)+y(t) x'(t)^3=0 \tag{2} $$
Looking at this, probably your best bet is to solve them numerically