According to this definition, a transformation is a map from set $X$ to itself. Does this mean they have to have same dimension?
What about transformation matrix? It is a mapping that does not have to have same dimension.
Do the "transformation" of the two definitions refer to the same transformation?
To answer your question, no, these are not the same. This is a case of using the same word for different concepts.
By your usage of the word "dimension", I gather you're thinking of vector spaces. The object $X$ in the first article you linked need not be have any additional, in particular it need not be a vector space. It is just a set, nothing more. With this (admittedly rather unfortunate) terminology, a transformation is just a map $T:X\to X$. You may picture it as moving around things in $X$.
On the other hand, given vector spaces $V$ and $W$, a function $T:V\to W$ is said to be a linear transformation if it satisfies $T(v+v')=T(v)+T(v')$ and $T(cv)=cT(v)$. When the (vector) spaces $V$ and $W$ are finite dimensional, then to $T$ we associate a matrix which encodes, in a certain way, all the relevant information about the object $T$. To reiterate, this only makes sense when the object $T$ is a linear transformation, so in particular $V$ and $W$ have to be vector spaces. It is not necessary for $\dim V=\dim W$. When they do have the same dimension, then the matrix will be square. It might be convenient to mention that, when $V=W$, then a linear transformation from $V$ to $W$ is sometimes called a linear operator.
Given an arbitrary set $X$ and a function $f:X\to X$, there is no canonical way of associating to $f$ a matrix which gives any information, because $X$ is just a collection of things, and what makes the above go through is the additional structure on the (vector) spaces, together with the linearity of the transformation.
It may be relevant to quote J. P. Serre,