Consider $GF(2^8)$ with reducing polynomial $m_p = x^8+x^4+x^2+x^1+x^0$, compute multiplication between $a=x^7+x^0$ and $b=x$.
Following https://en.wikipedia.org/wiki/Finite_field_arithmetic#Multiplication, it seems that the multiplication rule $a \cdot b = (a\cdot b) \mod m_p$. However, if we use this rule, we have $(x^7+x^0)\cdot(x^1) \mod (x^8+x^4+x^2+x^1+x^0)$, this results in $x^8+x^1$, which is beyond the Galois field. Is this multiplication rule wrong? If so, what is the formulated rule for finite field multiplication?
As $GF(2^8)$ is a field it satisfies field axioms, meaning that between others, is closed by multiplication.
You've said that multiplication is reduced modulo an irreducible polynomial over $F_2$. This is right. But seems that you failed on performing the operation as $(x^7+1)x \equiv x^4+x^2+1 \pmod{x^8+x^4+x^2+x+1}$. Moreover $x^8+x^4+x^2+x+1$ is not irreducible over $F_2$, so the structure is not $GF(2^8)$ but an univariate polynomial quotient ring $\frac{F_2[X]}{x^8+x^4+x^2+x+1}$.