I have this $3\times 3$ square above, where the $5$ white squares form a board $B$, and I am trying to calculating the rook polynomial of $B$, using the following formula :
the answer is given as $r_b(x)$=$1+5x+6x^2 +x^3$ but I don't understand how this is obtained from the formula as when I try I do not obtain that?
The given formula is the rook polynomial for a $m\times n$ board without restrictions and it is not useful here (for a $3\times 3$ board it is $1+9x+18x^2+6x^3$).
On the other hand, recall that, by definition, the rook polynomial of a board $B$ is $$R_B(x)=\sum_{k\geq 0}r_kx^k$$ where $r_k$ is the number of ways to place $k$ non-attacking rooks on the board $B$.
In our case, since the given board is $3\times 3$, the degree of the polynomial is $\leq 3$. Now $r_0=1$, $r_1=5$ (the number of available boxes), $r_2=6$ i.e. $2$ non-attacking rooks can be placed at $$(1A,2C), (1A,3C), (2A,1B),(2A,3C), (1B,2C), (1B,3C),$$ and $r_3=1$ i.e. the unique arrangement for $3$ non-attacking rooks is $(2A,1B,3C)$.
Therefore we find that the rook polynomial is $1+5x+6x^2+x^3.$