I am working on #3 of this problem set:
https://www.cpp.edu/~hspc/problems/HSPC_2019_Middle_School_Questions.pdf
and I wanted to try solving it numerically. My approach was this:
number of mats = total different combinations of RGB - (number of those combinations that fail the column condition + number of combinations that fail the row conditions - any overlap between the 2)
to get the total number of different RGB solutions (even those that fail the constraint) I think (3n)^3 works. However, I don't know where to go after this, how to find the number of combinations that will fail the constraint.
Your general approach of meeting constraints separately and then correcting for the overlap is a good one. A systematic way of applying it is the principle of inclusion–exclusion.
However, in the present case the solution is easier if you deal with the second constraint first and directly build it into the count.
Your count of $(3n)^3$ for the unconstrained case should be the other way around, $3^{3n}$. If you have $2$ cereals to choose from every morning, then over the course of three days you have $2^3=8$ choices, not $3^2=9$.
To satisfy the second constraint, note that there are $3^3=27$ different configurations of each column, and $3$ of these are monochromatic. Thus there are $27-3=24$ admissible configurations per column. So instead of $3^{3n}=\left(3^3\right)^n=27^n$, we can start with a count of $24^n$, and then we only have to satisfy the first constraint.
This is where inclusion–exclusion comes in. There are three conditions, one for each row, and we want to count the mats that violate none of them. To do that, we need to first count the mats that violate $k$ particular conditions.
There are $24$ mats that violate all three conditions, namely the $24$ admissible column patterns repeated across all columns.
If a mat violates two particular conditions, there are two monochromatic rows. If these rows have the same colour, there are $3$ choices for that colour, and then there are $2$ choices for each cell of the remaining row (since they can’t have the same colour as the two monochromatic rows). If the two monochromatic rows have different colours, there are $3\cdot2=6$ choices for those colours, and then the cells of the remaining row can have any of $3$ colours (since the two monochromatic rows with different colours already guarantee that no column is monochromatic). Thus in total the contribution from violating two particular conditions is $3\cdot2^n+6\cdot3^n$.
If a mat violates one particular condition, there are $3$ choices for the colour, and each column of the remaining two rows can have any of $3^2-1=8$ patterns (since it mustn't have both cells the colour of the monochromatic row). Thus the contribution from violating one particular condition is $3\cdot8^n$.
Adding this all together with the appropriate alternating signs and the binomial coefficients $\binom3k$ for choosing the $k$ particular conditions violated, we end up with a count of
$$ \binom3024^n-\binom318^n+\binom32\left(3\cdot2^n+6\cdot3^n\right)-\binom33\cdot24\\[15pt] =24^n-3\cdot8^n+18\cdot3^n+9\cdot2^n-24\;. $$