I tried to find the extended form of the group generated by the following operators.
(I): The first operator $$A=z\frac{\partial }{\partial z}+1$$ To find the extended form of the group generated by $A$, we must solve the following equations $$\frac{\partial z(a')}{\partial a'}=z(a')$$ and $$\frac{\partial v(a')}{\partial a'}=v(a')$$ where $z(0)=z$ and $v(0)=1$
Now from the first equation we get $$\int \frac{\partial z(a')}{\partial a'} \frac {1}{z(a')}da'= \int d a' $$ $$ \log z(a') = a' +k $$ we put $a'=0$ to find the constant $k$ $$ \log z = k $$ Then $$ \log z(a') = a' + \log z $$ $$ z(a') = z e^{a'}$$
On the other hand from the second equation we get $$\int \frac{\partial v(a')}{\partial a'} \frac {1}{v(a')}da'= \int d a' $$ $$ \log v(a') = a' +k $$ we put $a'=0$ to find the constant $k$ $$ k=0 $$ Then $$ \log v(a') = a' $$ $$ v(a') = e^{a'}$$
Thus $$ e^{a' A} f(x,y,z)=e^{a'}f(x,y,ze^{a'}) $$
(II): The second operator $$B=\frac{(x^2 - ay)}{yz}\frac{\partial }{\partial x}-\frac{x}{y}\frac{\partial }{\partial z}$$
To find the extended form of the group generated by $B$, we must solve the following equations
$$\frac{\partial x(b')}{\partial b'}=\frac{(x^2(b')-ay)}{yz(b')}$$ and $$\frac{\partial z(b')}{\partial b'}=- \frac{x(b')}{y}$$ where $z(0)=z$ and $x(0)=x$
This equations need to be solved, but I can't do anything after this step.
I have the final answer but i want to understand how it result
This is the final answer $$ e^{b' B} f(x,y,z)=f(\frac{(xz-ab')\sqrt{ay}}{\sqrt{(xz-ab')^2 - z^2 (x^2 -ay) }},y,\frac{1}{\sqrt{ay}}\sqrt{(xz-ab')^2 - z^2 (x^2 -ay) }) $$
Any input would be helpful for me .
I gather your (I) and (II) are independent, beyond being 1-parameter Lie groups, so, all Lie advections being equivalent to translations, this is just a nasty exercise in untangling coordinates.
I'll just consider your advective flow (II). I'll skip the pesky prime in your Lie flow parameter b', and assume a and y are constant parameters --- they don't vary in the Lie flow. w.l.o.g. I set them =1, i.e., I absorb them in b and x.
With the PDEs now corrected (my edit, to comport with your generator B and the answer), it is evident the variable $\omega\equiv zx$ behaves more simply, so your propagate PDEs reduce to $$ \frac{\partial \omega(b)}{\partial b}=-1, \qquad \frac{\partial z(b)}{\partial b}=-\omega(b)/z(b). $$
Integrating the first,
$$ \omega(b)=xz-b=\omega(0)-b. $$ Substituting into the second, $$ \frac{\partial z(b)^2}{\partial b}= 2(b-\omega(0)) $$ so $$ z(b)^2=(b-xz)^2 +c. $$ The constant c is fixed by the initial condition b =0, $ c=z^2(1-x^2)$.
The Lie flow equation is then $$ f(xz,z)\mapsto f\left (xz-b, \sqrt{(b-xz)^2+z^2(1-x^2)}\right). $$ This is of course equivalent to your equation, once you reorganize your variables into xz and z, and absorb a and y to unity, as described.