Let $A$ be a matrix where the columns form an orthonormal basis and let $\phi (v)=Av$.
I want to show that $\phi (x)\cdot \phi (y)=x\cdot y$ for all $x,y\in \mathbb{R}^3$.
Since the columns of $A$ form an orthonormal basis that means that $A^TA=I$, or not?
So let $\phi (x)\cdot \phi (y)=B$ then we have the following: $$A\cdot x\cdot A\cdot y=B\Rightarrow A^TA\cdot x\cdot A\cdot y=A^TB\Rightarrow x\cdot A\cdot y=A^TB\Rightarrow x\cdot A\cdot y=A^TB$$ Is that the correct way? But how do we continue?
You should probably try to be very careful with notation and not write "$\cdot$" for a matrix product.
Your calculation is correct for the very different quantity, $A(x^TAx) = A(x\cdot Ax),$ which is a matrix (not a scalar.) Only here is it possible to cancel $A$ by multiplication on the left (actually $x\cdot Ax$ is a scalar, so multiplying on the right also works): $$ B=A(x\cdot Ax) \implies A^TB = x\cdot Ax$$ In contrast, $Ax\cdot Ay=(Ax)\cdot(Ay)$ is a dot product of vectors, and hence a scalar.