There's a mapping $(x,y) \mapsto(u,v)$ given by
$u= x\cos\theta-y\sin\theta$
$v =x\sin\theta + y\cos\theta$
I'd like to find a generating function $G(x,y)$ for this mapping, which I understand to mean that
$\partial_x G = y$
$\partial_u G = v$
How does one do this?
I've tried the following:
\begin{align} G &= xy +f(y)\\
&= (u\cos\theta+v\sin\theta)(-u\sin\theta + v\cos\theta) +f(y)\\
& = -u^2\sin\theta \cos\theta+uv\cos^2\theta-uv\sin^2\theta -v^2\sin\theta \cos\theta +f(y)
\end{align}
$\partial_u G = -2u\sin\theta \cos \theta +v \cos^2\theta-v\sin^2\theta+ \partial_u f(y)$
$v = -2u\sin\theta \cos \theta +v \cos^2\theta-v\sin^2\theta+ \partial_u f(y)$
$\partial_u f = 2v\sin^2\theta +2u\sin\theta\cos\theta$
$ f = 2vu\sin^2\theta +u^2\sin\theta\cos\theta$
$G = xy +2vu\sin^2\theta +u^2\sin\theta\cos\theta$
$G = xy +2(x\sin\theta + y\cos\theta)(x\cos\theta-y\sin\theta)\sin^2\theta +(x\cos\theta-y\sin\theta)^2\sin\theta\cos\theta$
I then try to take the partial derivative of $G$ w.r.t. $x$ and see that I recover $y$. I don't think I do, but things get so tangled I may have made a mistake in arithmetic.
(This question comes from The Mathematical Mechanic by Mark Levi.)
I will slightly modify the notation and write $(p,q,P,Q)$ for $(x,y,u,v)$. This way you are looking for the generating function $G(x,u)=F_4(p,P)$ and the relevant equations are actually $$q(p,P)=-G_p(p,P),\:Q(p,P)=G_P(p,P).$$ I'm writing everything explicitly because my own mechanics courses (and mainly my own errors) have made me appreciate such an approach! You've made such an error at the beginning of your calculation, when you integrated $$G_x=y\,\text{ to }\,G=xy+f(y),$$ when in fact $y$ (or in my case, $q$) is a function of $x$ and $u$. I urge you to stop reading this answer now and have another go at it! In any case, here is the full solution.
First we solve for $(q,Q):$
$$q=p \cot (\theta ) -P\csc (\theta )\\ Q=p \csc(\theta)-P \cot (\theta )$$ Next we integrate $q=-G_p$: $$\frac{1}{2} p^2\cot(\theta)-pP\csc(\theta)+f(P)=-G$$ Differentiate wrt. $P$: $$G_p = +p\csc(\theta)-f'(P)=Q(p,P)=p \csc(\theta)-P \cot (\theta )$$ Giving us $$f(P)=\int\mathrm dP f'(P)=\int\mathrm dP P \cot (\theta )=\frac{1}{2} P^2\cot(\theta)$$ So that $$G=-\frac{1}{2} p^2\cot(\theta)+pP\csc(\theta)-\frac{1}{2} P^2\cot(\theta).$$ You can check that this is correct.