Given a presentation $\langle x, y \mid x^n = y^2 = 1, xy = yx^2 \rangle$, we know that there is a implicit relation, since $x = xy^2 = (xy)y = (yx^2)y = yxyx^2 = y^2x^4 = x^4$, thus $x^3 = 1$, this shows that the order of $x$ is less than or equal to $3$.
My question is if we let $n = 3k$, what would be the order of $x$? Under this condition can we conclude that order of $x$ is equal to $3$?
On
We can "straighten" any word in $x$'s and $y$'s into the form $y^a x^b$ by moving any $y$'s to the right of $x$'s left. As you've noted, $0 \leq a \leq 1$ and $0 \leq b \leq 2$, so the group has size 6 at most. Indeed, your group with $n=3$ is one of the usual presentations of the dihedral group of order $6$, where $x$ is the order-three rotation of a regular triangle and $y$ is reflection. The same clearly holds when $n=3k$. When $n=2$, your group should collapse down to the cyclic group of order $2$, $\langle y \mid y^2 = 1\rangle$.
Yes, the order of $x$ is $3$ if and only if $n$ is a multiple of $3$.
First, if the order of $x$ is $3$, the fact that $x^n = 1$ implies that $3$ divides $n$.
Now, suppose that $n$ is a multiple of $3$. You showed that $x^3 = 1$ in this group, so the order of $x$ divides $3$ – we must only show that the order of $x$ is not $1$ (i.e. that $x \neq 1$) to conclude that the order of $x$ is $3$. To see this, note that by universal property, there exists a homomorphism from this group to $S_3$ which sends $x$ to $(1 \; 2 \; 3)$ and $y$ to $(1 \; 2)$. In particular, $x$ is not sent to the identity element of $S_3$, so $x \neq 1$, as desired.
Edit: I'll include a quick description of the universal property I mentioned.
Let $G = \langle X \mid R \rangle$ – $X$ is the set of generators in the presentation, and $R$ is the set of relations in the presentation (every equational relation in a group presentation can be written in the form $\text{something} = 1$, and it is this $\text{something}$ that we call the relation and put in the set $R$). In your example, $X = \{x,y\}$ and $R = \{x^n, y^2, xyx^{-2}y^{-1}\}$. Let $F = \langle X \mid \varnothing \rangle$ be the group presentation with the same set of generators and no relations, a.k.a. the free group on the set $X$.
Theorem 1. (Universal Property of Free Groups)
For each group $H$, each function $f : X \to H$ extends to a unique homomorphism $\overline{f} : F \to H$.
This is exactly like how a linear map between vector spaces is uniquely determined by what it does to a basis – once you decide on a function (any function) from the basis to the codomain, it extends uniquely to a linear map (a.k.a. vector space homomorphism) from the domain. This is actually quite easy to prove – you should try to do so, and even if you don't it's important to think through this and understand how $\overline{f}$ must be constructed (or else you will be lost later on). Now we can generalize this to a similar statement about all group presentations:
Theorem 2. (Universal Property of Group Presentations)
For each group $H$, a function $f : X \to H$ extends to a unique homomorphism $G \to H$ if and only if $\overline{f}(r) = 1$ for all $r \in R$.
As mentioned previously, Theorem 1 is a special case of Theorem 2 where $R = \varnothing$, although (at least the way I wrote it) we needed the result of Theorem 1 to write Theorem 2 concisely. This is harder to prove, because you have to be very careful about defining the map $G \to H$ and proving it's a homomorphism. If you want some practice proving something nontrivial directly from the definition of "group presentation", this is a good exercise. Anyway, we can apply this in your example to produce a map from your group to $S_3$:
Let $f : \{x, y\} \to S_3$ be the function which sends $x$ to $(1 \; 2 \; 3)$ and $y$ to $(1 \; 2)$. We see that $$\overline{f}(x^n) = (1 \; 2 \; 3)^n = ((1 \; 2 \; 3)^3)^k = 1^k = 1,$$ $$\overline{f}(y^2) = (1 \; 2)^2 = 1,$$ $$\overline{f}(xyx^{-2}y^{-1}) = (1 \; 2 \; 3) (1 \; 2) (1 \; 2 \; 3)^{-2} (1 \; 2)^{-1} = (1 \; 2 \; 3) (1 \; 2) (1 \; 2 \; 3) (1 \; 2) = 1.$$ Thus, there is a unique homomorphism $\langle x, y \mid x^n = y^2 = xyx^{-2}y^{-1} = 1 \rangle \to S_3$ extending $f$.