Hi,
For this problem my approach is obviously lacking, as a I do not utilize the hint. But I am not sure how to utilize the hint or why it is necessary to do this problem.
This is what I did:
$$ \langle Ax, Ay \rangle = \langle x,y\rangle $$
$$ A\langle x,y\rangle = \langle x,y\rangle $$
$$ \|A\langle x,y\rangle\| = \| \langle x,y\rangle \|$$
The above only holds if $\|A\| = 1$, which is true with $A$ is an orthogonal transformation.
Thus,
$$ \|Ax \| = \|x\| $$
The proof is split in two parts, one for each direction. The first one is easy.
Show: $(Ax,Ay)=(x,y) \Rightarrow ||Ax||=||x||$: $$ ||Ax||^2=(Ax,Ax)=(x,x)=||x||^2 $$ The second part, the $\Leftarrow$ direction: $$ ||x+y||^2=||A(x+y)||^2=\langle A(x+y),A(x+y)\rangle=||Ax||^2+||Ay||^2+2\langle Ax,Ay\rangle $$ and $$ ||x-y||^2=||A(x-y)||^2=\langle A(x-y),A(x-y)\rangle=||Ax||^2+||Ay||^2-2\langle Ax,Ay\rangle $$ subtracting: $$ ||x+y||^2-||x-y||^2=4\langle Ax,Ay\rangle $$ Finally, using the given identitiy: $$ \langle Ax,Ay\rangle=\frac{1}{4}(||x+y||^2-||x-y||^2)=\langle x,y\rangle $$
Remark: The basic idea is to express the scalar product $\langle Ax,Ay\rangle$ in terms of $||x+y||^2$ norms (plus and minus) and then using the given identity.