Consider the functional $S[y]=\int_{1}^{2}ln(1+x^2y')dx, y(1)=0, y(2)=A$, where $A$ is a constant and $y$ is a continuously differentiable function for $1\leq x\leq 2$. Let $h$ be a continuously differentiable function for $1\leq x\leq 2$, and let $\epsilon$ be a constant. Let $\bigtriangleup =S[y+\epsilon h]-S[y]=\epsilon\int_{1}^{2}\frac{x^2h'}{1+x^2y'}dx-\frac{\epsilon^2}{2}\int_{1}^{2}\frac{x^4h'^2}{(1+x^2y')^2}dx+O(\epsilon^3)$.
a) Show that if $h(1)=h(2)=0$, then the term $O(\epsilon)$ in this expansion vanishes if $y'(x)$ satisfies the equation $\frac{dy}{dx}=\frac{1}{c}-\frac{1}{x^2}$, where $c$ is a nonzero constant. Solve this equation to show that the stationary path is $y(x)=\frac{x(1+2A)-(3+2A)}{2}+\frac{1}{x}$. For what range of values of $A$ is this derivation valid?
b) By examining the value of the $O(\epsilon^2)$ term in $\bigtriangleup$, determine whether $S[y]$ has a local maximum or minimum on the stationary path.
Here's my work for part a):
Suppose $y'(x)$ satisfies the equation $\frac{dy}{dx}=\frac{1}{c}-\frac{1}{x^2}$, where $c$ is a nonzero constant.
Then $\bigtriangleup =\epsilon\int_{1}^{2}\frac{x^2h'}{1+x^2(\frac{1}{c}-\frac{1}{x^2})}dx-\frac{\epsilon^2}{2}\int_{1}^{2}\frac{x^4h'^2}{(1+x^2(\frac{1}{c}-\frac{1}{x^2}))^2}dx+O(\epsilon^3)=\epsilon\int_{1}^{2}(ch')dx-\frac{\epsilon^2}{2}\int_{1}^{2}(c^2h'^2)dx+O(\epsilon^3)$.
Consider the term $O(\epsilon)$ in this expansion.
Let $h(1)=h(2)=0$.
Observe that $\epsilon\int_{1}^{2}(ch')dx=\epsilon[ch]_{1}^{2}=\epsilon(c(h(2)-h(1)))=0$.
Thus, the term $O(\epsilon)$ in this expansion vanishes if $y'(x)$ satisfies the equation $\frac{dy}{dx}=\frac{1}{c}-\frac{1}{x^2}$, where $c$ is a nonzero constant.
To solve this equation, we integrate this equation to get $y(x)=\frac{x}{c}+\frac{1}{x}+k$ where $k=constant$.
This means $y(1)=0\implies 0=\frac{1}{c}+1+k$ and $y(2)=A\implies A=\frac{2}{c}+\frac{1}{2}+k$.
Solving these two equations, we have $k=-\frac{3}{2}-A$ and $c=\frac{2}{3+2A-2}$.
Thus $y(x)=\frac{x(3+2A-2)}{2}+\frac{1}{x}+(-\frac{3}{2}-A)$.
Hence, the stationary path is $y(x)=\frac{x(1+2A)-(3+2A)}{2}+\frac{1}{x}$.
From here, I'm stucked. How should I find out the range of values of $A$ that this derivation is valid? Also, for part b) of this problem, I know that we have to consider $-\frac{\epsilon^2}{2}\int_{1}^{2}\frac{x^4h'^2}{(1+x^2y')^2}dx$ but by examining this, how should we determine whether $S[y]$ has a local maximum or minimum on the stationary path?
a) $S[y]$ must be well-defined when $y$ is the stationary path. In particular, the argument of the logarithm in the integrand of $S$, $1+x^2y'$, must be positive. Since the stationary path satisfies $y'=\frac{1}{c}-\frac{1}{x^2}$, that implies $c\left(=\frac{2}{1+2A}\right)>0$, or $A>-\frac{1}{2}$.
b) When $y$ is the stationary path, $S[y+\epsilon h]-S[y]=-\frac{\epsilon^2}{2}\int_{1}^{2}(c^2h'^2)\,dx+O(\epsilon^3)\leq 0$ for sufficiently small $\epsilon$. This implies that the stationary path is a local maximum of $S$.