I'm trying to understand the concept of mixing in dynamical systems theory, especially when the system in question has a measure-preserving flow. Here's how the condition is expressed mathematically: If $\mu$ is the measure and $\phi$ is the flow, then for all subsets $A$ and $B$ of positive measure, $\lim_{t \rightarrow \infty}\mu(\phi^{t}(B) \cap A) = \mu(B) \times \mu(A)$.
Now suppose $B$ is an arbitrary set with measure greater than 0 and less than 1. If the flow is measure preserving, then for all $t$, $\mu(\phi^{t}(B)) = \mu(B)$. Pick $A = \lim_{t \rightarrow \infty}\phi^{t}(B)$. Then, $\mu(A) = \mu(B)$. It follows that $\lim_{t \rightarrow \infty}\mu(\phi^{t}(B) \cap A) = \mu(A) = \mu(B)$.
So if the dynamics is mixing, then we will have $\mu(B) = \mu(B) \times \mu(B)$. But this is only possible if $\mu(B)$ is 0 or 1, contradicting our initial assumption.
Isn't this a problem with the definition of mixing? Is the definition in my source wrong? Or am I doing something wrong?
There seems to be suspicious reasoning as pointed out in the comments when you define $A$ and then conclude that since $\mu(A)=\mu(B)$ then $\lim_{t\rightarrow\infty}\mu(\phi^t(B)\cap A)=\mu(A)$.
Have you encountered ergodic theory? Your mixing condition is more formally referred to as "strong mixing" which implies ergodicity, in that $\phi_t$ must be ergodic. By definition a flow is ergodic when $A=\phi_{-t}(A)$ implies $\mu(A)=0,1$.