Suppose we have a functional of the form $$ J(y)= \int_{x_{0}}^{x_{1}} f(x,y,y')dx $$ and a smooth transformation $X=\theta (x,y: \epsilon)$ , $Y=\psi (x,y: \epsilon)$ where $\epsilon$ is a parameter. I have the following definition of invariance under a transformation:
Definition. The functional $J$ is invariant under the above transformation if, for all $\epsilon$ sufficiently small, in any subinterval $[a,b] \subset [x_{0},x_{1}]$ we have that $$ \int_{a}^{b} f(x,y,y')dx= \displaystyle \int_{a_{\epsilon}}^{b_{\epsilon}} f(X,Y,Y')dX $$ for all smooth functions $y$ defined on $[a,b]$. Here, $a_{\epsilon}=\theta (a,y(a): \epsilon)$, $b_{\epsilon}=\theta (b,y(b): \epsilon)$
Once said that, I'm interested in proving the invariance of the functional $$ J(y)=\int_{x_{0}}^{x_{1}} xy'^2dx $$ under the transformation $$ \begin{cases} X=x+2\epsilon x \ln(x)\\ Y=(1+\epsilon)y \end{cases} $$
My attempt:
This would be easier if we could solve explicitly for $x$ in the first equation of the transformation, but, since we can't, suppose $x=\Theta (X; \epsilon)$ such inverse function exist(locally and for $\epsilon$ sufficiently small because of the inverse function theorem). On the other hand, we easily get $y=1/(1+\epsilon) Y$. Therefore, $$ y'(x)=\displaystyle \frac{dy}{dx}=\displaystyle \frac{Y'(X)}{(1+ \epsilon) \Theta'(X)} $$ which implies that $$ xy'dx= \frac{\Theta(X)}{\Theta '(X)} \frac{Y'(X)^2}{(1+\epsilon)^2}dX $$ So, as you can see, if I could prove $$ \frac{\Theta(X)}{(1+\epsilon)^2 \Theta '(X)}=X $$ I'd be done, but I could not prove it, and actually, it doesn't seem true.
Any help? Thanks in advance.
Update:
Of course we can also try to express $XY'(X)^2 dX$ in terms of $x$ and $y'(x)$. In this case, we get $$ XY'(X)^2 dX= \displaystyle \frac{x+2 \epsilon x ln(x)}{1+ 2 \epsilon ln(x)+2 \epsilon} (1+ \epsilon)^2 y'(x)^2 dx $$ In any case, it is not clear to me how to reconcile this expressions with the definition :( .
Actually, this transformation is NOT a variational symmetry. From your computations above we have that, $$ \displaystyle \int_{a_{\epsilon}}^{b_{\epsilon}} XY'(X)dX= \displaystyle \int_a^b xy'^2 dx + \displaystyle \int_a^b \displaystyle \frac{y'^2(x \epsilon ^2+2 \epsilon ^3 x ln(x))}{1+2 \epsilon ln(x)+2 \epsilon} dx $$ And this definitely does not agree with the definition. On the other hand, even if we ignore this, if this transformation were a variational symmetry, because of the definition we should have that $$ \displaystyle \frac{d}{d \epsilon} \left( \displaystyle \int_{a_{\epsilon}}^{b_{\epsilon}} f(X,Y,Y') dX \right)=0 $$ But, in this case, this is obviously false.
Remark:
Althought this is not a variational symmetry, it's certainly true that the functional is infinitesimally invariant under the transformation, but it's no the same thing, check the definition in the book Advanced Engineering Analysis by Leonid P. Lebedev.