I have found a (In how many ways can 4 red balls and 7 blue balls be arranged in 3 boxes)(question).
The question says:
In how many ways can 4 red balls and 7 blue balls be arranged in 3 boxes where each box must contain at least 1 red ball and each box can contain less than or equal or 4 balls, under the following cases?
(1) Each ball is distinct and Each box is distinct
(2) Each ball is distinct and Each box is identical
For the first part I have come up with two distinct solutions:
First solution:
Distribute the red balls such that select which box will have two red balls and distribute others: $$\binom{4}{2}\times 3 \times 2$$
Now we have 7 distinct balls to distribute. The box containing two red balls can have at most two balls, and the other box can have maximum three balls. Then use exponential generating functions to disperse them: $$7! \times [x^7]\bigg(1+x+\frac{x^2}{2!}\bigg)\bigg(1+x+\frac{x^2}{2!}+\frac{x^3}{3!}\bigg)^2$$ $$36 \times \frac{7!}{9}=20160$$
Second way: Give one red ball to each boxes such that $$4 \times 3 \times 2 =24$$
Now, we have $8$ distinct ball to be distributed where each box can contain at most $3$ ball now. So $$8![x^8]\bigg(1+x+\frac{x^2}{2!}+\frac{x^3}{3!}\bigg)^3$$ $$24 \times \frac{8!}{24}=40320$$
Which answer is correct ? There are some other solutions in given links. PLease explain if my answers is wrong.
NOTE: For the second part , I thought dividing the result by $3!$
$Q1$
1: Your first answer is correct.
[2]: Your second answer overcounts, (in fact exactly double counts) because you have put one red ball in each box first. For the double red box, R1 initially and R3 later is the same as putting R3 first and R1 later
$Q2$
For the second part, there is a subtlety, we can't straight away divide by $3!$ . As requested, I am giving a non-G.F. answer to clarify.
Non-G.F. answer
There are two possible configurations
$2R\;1B\;\;1R\;3B\;\;1R\;3B$, with $3$ permutations, and
$2R\;2B\;\;1R\;2B\;\;1R\;3B$, with $6$ permutations.
So for part $1$, the computation is
$3\times\binom42\binom71\binom21\binom63 + 6\times\binom42\binom72\binom21\binom52 = 20160$
while for part $2$, the computation is $\;\;\binom42\binom71\binom21\binom63\div 2! + \binom42\binom72\binom21\binom52 = 3360$
You could have a look here on team formation