The book gave the answer as $12$.
This is how I approached it:
We need the arrangements in the form
math - physics - math - physics - math
or else two math books will be adjacent. So, we choose $2$ positions for the two physics books to be placed and multiply it by $2!$, since either position can have either book. Next, we see that there are three remaining spaces for the math books, so we multiple by $3!$ giving us:
$$ 2 \times 2 \times 3 \times 2 \times 1 $$
However, since we chose the positions once for the physics books, and then another for the math books, we must divide by $2$ so we don't double count. Does this seem correct?
I think you overcomplicated it a little bit.
Your argument that the arrangement should be
$$ MPMPM $$
is correct, $M$ standing for math books and $P$ standing for physics.
Now you could basically look at the number of permutations between the $3$ math and $2$ physics books. Which gives
$$ 3!\cdot 2!=12 $$
different arrangements.
You can think of this last step as "we know that the books will be in this order $MPMPM$" and this is not changed by swapping two math books for example.
We can in a way decompose this into two parts
$$ M\ \ M\ \ M $$ and $$ P\ \ P $$
and look at the number of ways the individual parts can be arranged and then using product rule to reassamble them.
P.S: I don´t really understand your last paragraph about overcounting. Would you elaborate on it a bit?