I know that the only groups of order 4 are $\mathbb{Z}_2\times \mathbb{Z}_2$ and $\mathbb{Z}_4$ up to isomorphisms. And I also know that the group presentation of $\mathbb{Z}_4$ is $\left ( a:a^4=1 \right )$.
Naturally, I thought the group presentation of $\mathbb{Z}_2\times \mathbb{Z}_2$ is $\left ( a,b:a^2=1,b^2=1 \right )$. But I'm not sure this presentation really indicates $\mathbb{Z}_2\times \mathbb{Z}_2$.
For example, if $A=\begin{pmatrix} \sqrt{2} &1 \\ -1 &-\sqrt{2} \end{pmatrix}, B=\begin{pmatrix} \sqrt{3} &1 \\ -2 &-\sqrt{3} \end{pmatrix}\in \left ( M(2, \mathbb{R}), \cdot \right )$, then a group $\left ( A,B \right )$ has a presentation $\left ( A,B:A^2=E,B^2=E \right )$ but $AB\neq BA$ and thus this group is not isomorphic to $\mathbb{Z}_2\times \mathbb{Z}_2$.
Where did I misunderstand?
You are correct that $\langle a, b | a^2 = b^2 = 1\rangle$ is not enough to specify the given group: In fact, this is the large non-abelian group $\mathbb{Z}_2 \ast \mathbb{Z}_2$, where $\ast$ is the free product. Its elements can be thought of as sequences of various forms: $$abab\dots abab, baba\dots baba, abab\dots aba, baba\dots bab$$ So it's necessary to add a bit more; the easiest way would be to modify the definition to add a commutativity relation:
$$\langle a, b | a^2 = b^2 = 1, ab = ba\rangle$$
Then one can rearrange the given sequences, finding that all elements are of the form $a^k b^j$; then there are exactly four group elements, corresponding to the possible parities of $k$ and $j$.
There are, of course, other ways to represent this group.