We know that if $R\colon A\to B$, we can define the inverse relation as follows: $$R^{-1}=\{(y,x)\in B\times A\mid(x,y)\in R\}.$$ Now I want to know if this set, let's call it $R\:'^{-1}$, is the same as $R^{-1}$: $$R\:'^{-1}=\{(x,y)\in R\mid(y,x)\in B\times A\}.$$ I think $R^{-1}\neq R\:'^{-1}$ so I picked an example.
I tried with $A=\{1,2\}$, $B=\{3,4\}$, so $B\times A=\{(3,1),(3,2),(4,1),(4,2)\}$. Let's pick $R=\{(3,1),(4,1)\}$. Then, by definition, $$R^{-1}=\{(1,3),(1,4)\}.$$ But if we look at the definition of $R\:'^{-1}$, I think it says "All the elements of $(x,y)\in R$ such that the pair $(y,x)$ is in $B\times A$". Hence, $R\:'^{-1}=\emptyset$, because we know, for example, that $R$ has $x=3$ and $y=1$, so $(3,1)\in R$, but this proposition: $(y,x)=(1,3)\in B\times A$ is false. Thus $R^{-1}\neq R\:'^{-1}$.
Are my understanding of $R\:'^{-1}$ and counterexample correct?
In your counterexample, your choice of $R$ is wrong. You picked $R=\{(3,1),(4,1)\}\subseteq B\times A$ when $R$ is supposed to be a subset of $A\times B$.
Coming back to your definition of $R^{\prime-1}:=\{(x,y)\in R\mid(y,x)\in B\times A\}$. By your definition, $R^{\prime-1}$ is a subset of $R$, and in fact is equal to $R$, since any element $(x,y)\in R$ satisfies $(y,x)\in B\times A$.
I think the concept that you are working around is the fact that $R$ and $R^{-1}$ are in bijection with each other. Indeed, we have the bijection $$\varphi:\left\{\begin{array}{ccc} R & \rightarrow & R^{-1} \\ (x,y) & \mapsto & (y,x) \end{array}\right.$$ So in a sense, $R^{-1}$ is "equivalent" to $R$, except that the coordinates are flipped. I'm just rambling now but I hope this proves helpful.