I am confused with the definition of ordinals: "A set $a$ is an ordinal if it is transitive and totally ordered by $\in$." Then an ordinal is a set but we also know that all elements of ordinals are ordinals themselves.
If we take $\alpha=\{x,\{x\}\}$ then $\alpha$ is a transitive set and totally ordered by $\in$ so it is an ordinal. But $x\in\alpha$ so $x$ is an ordinal but $x$ is not a set and ordinals are sets.
Can someone please fill in this gap in my understanding?
A set is an ordinal if it's transitive and well-ordered with respect to $\in$.
Well-ordered means that every non-empty subset has a least element. Transitive means that every element is also a subset.
What this means for your example is that $\alpha = \{ x, \{ x \}\}$ is transitive if $x \subset \alpha$. The only case where $\alpha$ is transitive is if $x = \emptyset$, in all other cases $\alpha$ is not actually an ordinal because then you don't have $x \subset \alpha$.
Edit
The ordinals are an extension of the natural numbers, see here, and the natural numbers start at $0$ which is the empty set $\emptyset$. So every ordinal has to contain $0$, that is, the empty set.
Hope this helps.