Two undistinguishable letters $a$ and two undistinguishable letters $b$ are filled into $16$ cells of a square matrix. It is required that each cell contains at most $1$ letter and each row or column cannot contain the same letters. What is the number of ways which the matrix can be filled in?
My attempt:
I considered the case of first placing the two '$a$' s in $\frac{\binom{16}{1}\binom{9}{1}}{2} =72$ ways and then the two '$b$'s in also $72$ ways.
Now we need to find the overlapping cases.
$(i)$
In the same cell we have got $72 +$ the case where one $b$ is in the same cell as that of $a$ and the other $b$ is in it's proper place. This is where I am getting confused.
Please help me out.
I think you've got a good idea. Let's say we start with $a$ and $b$ in a common cell, and place $a$ and $b$ freely in other cells that aren't in the same row/column, but also so that the second $a$ and $b$ end up in distinct cells. There are $16$ places to put the $a$ and $b$ together, then $9$ cells to place the second $a$, then $8$ cells to place the second $b$. In total, $$16 \times 9 \times 8 = 1152.$$ We also need to exclude the cases where both pairs of $a$ and $b$ end up together, which is simply $72$, as you calculated previously. So, the total should be $$72^2 - 1152 - 72 = 3960.$$