For $n \in \mathbb N$, finite group $V_{8n}$ is
$$ V_{8n} = \langle a, b : a^{2n} = e, b^4 =e, ba = a^{-1}b^{-1}, b^{-1}a = a^{-1}b \rangle. $$
This group is defined in this paper for all $n \in \mathbb N.$
I want to prove that $b^2 \notin \langle a \rangle$.
If possible, let $b^2 \in \langle a \rangle$. Then $b^2 = a^t$ for some $t$. I observe that the possibility of $t$ is $n$. Since $ba = a^{2n -1}b^3$ so that by mathematical induction on $i$, we get $$ ba^i = \left\{ \begin{array}{ll} a^{2n -i}b & \mbox{if $i$ is even };\\ a^{2n - i}b^3& \mbox{if $i$ is odd}\end{array} \right.$$
If $n$ is odd, then $b^3 = ba^n = a^n b^3$ implies $a^n = e$; a contradiction. I am stuck here if $n$ is even. I would be thankful for your kind help.
From the relations $bab=a^{-1}$ and $ba^{-1}b = a$, we get $b^2ab^2 = a$, and since $b^4=1$, this implies $b^{-2}ab^2 = a$, so $b^2 \in C_G(a)$. Since clearly $b^2 \in C_G(b)$, this gives $b^2 \in Z(V_{8n})$.
Now, the calculation in user1729's answer shows that $V_{8n}/\langle b^2 \rangle$ is isomorphic to the fihedral group of order $4n$. Since $b^4=1$, this proves that $|V_{8n}| \le 8n$ and $|V_{8n}| \ge 4n$. But to prove that $b^2 \not\in \langle a \rangle$, we still have to prove that $b^2 \ne 1$, which would show that $|V_{8n}| = 8n$.
To prove that, we follow the standard procedure of contructing a group $\langle a,b \rangle$ of order $8n$ in which $a$ and $b$ satisfy the relations of $V_{8n}$.
Start with the group $H = \langle a,c \mid a^{2n}=c^2=1, ac=ca \rangle \cong C_{2n} \times C_2$. Then $H$ has an automorphism $\phi$ of order $2$ with $\phi(a) = a^{-1}c$ and $\phi(c)=c$. Let $\langle b \rangle$ with $b$ of order $4$ be cyclic, and let $X$ be the semidirect product $H \rtimes_\phi \langle b \rangle$. So $$X = \langle a,b,c \mid a^{2n}=b^4=c^2=1, ac=ca, b^{-1}ab=a^{-1}c,b^{-1}cb=c \rangle,$$ and $|X| = 4|H| = 16n$.
Now $b^2, c \in Z(X)$, and we define $G = X/\langle b^2c^{-1} \rangle$, so $|G| = 8n$ and $$G = \langle a,b,c \mid a^{2n}=b^4=c^2=1, b^2=c, ac=ca, b^{-1}ab=a^{-1}c,b^{-1}cb=c \rangle \cong \langle a,b\mid a^{2n}=b^4=1, ab^2=b^2a, b^{-1}ab=a^{-1}b^2 \rangle.$$
Note that in $G$ we have $bab = b^2a^{-1}b^2 = a^{-1}$ (since $b^2 \in Z(G))$, and similarly $ba^{-1}b = a$, so the relations of $V_{8n}$ are indeed satisfied in $G$, and since $|G| = 8n$, this proves that $|V_{8n}| = 8n$ and $G \cong V_{8n}$.
Finally note that, in $G$, $b^2 = c \not\in \langle a \rangle$, which is what you wanted to prove.