Automorphisms of elliptic curves with a point of order N

Question

Probably a stupid question, but I'm trying to figure out the following: Suppose we have an element of $X_1(N)$, i.e. an elliptic curve $E$ with a point $p$ of order $N$. If we have another such curve $(E',p')$, then we say that they are isomorphic if there is an isomorphism of elliptic curves $\phi$ such that $\phi(E) = E'$ and $\phi(p) = p'$. Is it correct that an automorphism of these curves is an automorphism $\phi$ of $E$ such that $\phi(p) = p$? Or can we have $\phi(p)$ different from $p$. What worries me is that in the first definition it appears as if the curves don't have any automorphisms at all. Is this correct?