Example: $n = 3, a = 1, b = 3$. For clarity purposes, I will indicate balls placed in capacity $1$ trays within (), capacity $2$ trays within [], and capacity $3$ trays within {}.
For serving trays of capacity $1$, there are $6$ ways to fill them:
$(1), (2), (3)$
$(1), (3), (2)$
$(2), (1), (3)$
$(3), (1), (2)$
$(3), (2), (1)$
For serving trays of capacity $2$, there are $6$ ways to fill them:
$[1, 2], (3)$
$[1, 3], (2)$
$[2, 3], (1)$
$(1), [2, 3]$
$(2), [1, 3]$
$(3), [1, 2]$
For serving trays of capacity $3$, there is only 1 way to fill them:
$\{1, 2, 3\}$
This gives us a total of $x = 13$ ways. What I'm confused is how to go about calculating $x$ using the relevant combinations and permutations formulae (and factorials too) without having to manually list them out.
I was able to answer the questions with easy conditions such as $n = 6, a = 1, b = 1$, where the answer is $6!$, and others like $n = 8, a = 7, b = 7$, where the answer is $C(8,7)$.
However, one of the follow-ups had $n = 7, a = 3, b = 6$ and I tried my hand at it but failed to get the correct answer of $x = 238$. Could I get some explanations on how to tackle this question?
We'll assume $a\leq b$ so we don't have to keep writing $\max(a,b)$ and $\min(a,b).$
Let $F_{a,b}(n)$ be the number of ways to put $n$ distinct balls into trays of sizes from $a$ to $b,$ with the number of possible balls remaining is less than $a.$ Trays are more like bags, in that the order of the balls in a bag are not distinct.
Then we start with initial values:
$$F_{a,b}(n)=1, \text{ for }0\leq n<a,F_{a,b}(n)=0\text{ for } n<0$$
Then you get the following recursion:
$$F_{a,b}(n)=\sum_{k=a}^{b} \binom{n}{k}F_{a,b}(n-k), n\geq a.\tag1$$
We basically pick the first tray of size $k$ and then pick $k$ balls for the first tray, and then do the same process all over again with the remaining $n-k$ balls.
To compute $F_{3,6}(7),$ we compute:
$$\begin{align}F_{3,6}(3)&=1\\ F_{3,6}(4)&=\binom{4}{3}F_{3,6}(4-3)+\binom{4}{4}F_{3,6}(4-4)=5\\ F_{3,6}(5)&=\binom{5}{3}F_{3,6}(5-3)+\binom{5}{4}F_{3,6}(5-4)+\binom55F_{3,6}(5-5)=16\\ F_{3,6}(6)&=\binom{6}3\cdot 1+\binom{6}{4}\cdot 1+\binom{6}{5}\cdot 1+\binom{6}6\cdot 1=42\\ F_{3,6}(7)&=\binom{7}{3}\cdot 5+\binom{7}4\cdot 1+\binom75\cdot 1+\binom 76\cdot 1\\&=35\cdot5+35+21+7=238 \end{align}$$
If we let $G_{a,b}(n)=\frac{F_{a,b}(n)}{n!},$ your recursion becomes:
$$G_{a,b}(n)=\sum_{k=a}^{b} \frac1{k!}G_{a,b}(n-k)$$ which might be easier to calculate, since you only need to calculate $k!$ once for each $a\leq k\leq b.$ But you have to keep track of fractions.
But $G_{a,b}$ is also a simple homogeneous recurrence, so we have a lot of tools for this. In particular, $G_{a,b}$ is related to the roots of the polynomial:
$$x^b-\sum_{k=a}^b \frac1{k!}x^{b-k}.$$
For example, when $a=3,b=4$ you will have $G_{3,4}(n)\sim \alpha\cdot\beta^n$ for some constant $\alpha$ and $\beta\approx 0.61645$ as $n\to\infty.$
One approach is to consider the exponential generating function:
$$f_{a,b}(x)=\sum_{n=0}^{\infty}\frac{F_{a,b}(n)}{n!}x^n.$$
Then the recurrence (1) is equivalent to $$f_{a,b}(x)\sum_{k=a}^b\frac{x^k}{k!}=f_{a,b}(x)-\sum_{k=0}^{a-1}\frac{x^k}{k!}$$
of $$f_{a,b}(x)=\frac{\sum_{k=0}^{a-1} \frac{x^k}{k!}}{1-\sum_{k=a}^{b}\frac {x^k}{k!}}$$
When $a=b$ this gives $$f_{a,b}(x)=\left(\sum_{i=0}^{a-1}\frac{x^i}{i!}\right)\sum_{j=0}^{\infty}\frac{x^{aj}}{(a!)^j}$$
So the the coefficient of $x^n/n!$ in this power series is $$F_{a,a}(n)=\frac{n!}{(a!)^{\lfloor n /a\rfloor}(n-a\lfloor n/a\rfloor)!}$$
But the formula becomes messier when $a\neq b.$
The decision to look at exponential generating functions was due to the fact that if $$c_n=\sum_{k=0}^{n}\binom{n}{k}a_kb_{n-k},$$ then the exponential generating function of $c_n$ is the production of the exponential generating functions for $a_n$ and $b_n.$
In this case, $a_n=1$ for $a\leq n\leq b$ and $a_n=0$ otherwise. And $b_n=F_{a,b}(n).$
When $a=3,b=6$ the function becomes:
$$f_{3,6}(x)=\frac{1+x+x^2/2}{1-x^3/3!-x^4/4!-x^5/5!-x^6/6!}$$
that is fairly unwieldly.
The value $F_{3,6}(7)$ can be computed as $7!$ times the coefficient of $x^7$ in $$\left(1+x+x^2/2\right)\left(1+\left(\frac{x^3}{3!}+\frac{x^4}{4!}+\frac{x^5}{5!}+\frac{x^6}{6!}\right)+\left(\frac{x^3}{3!}+\frac{x^4}{4!}+\frac{x^5}{5!}+\frac{x^6}{6!}\right)^2+\cdots\right)$$
You can ignore the terms in the second half when the exponent is more than $7,$ so you get:
$$\left(1+x+x^2/2\right)\left(1+\frac{x^3}{3!}+\frac{x^4}{4!}+\frac{x^5}{5!}+\left(\frac1{6!}+\frac{1}{3!^2}\right)x^6+\frac{2x^7}{3!4!}\right)$$
So: $$F_{3,6}(7)=7!\left(\frac{2}{3!4!}+\frac{1}{6!}+\frac{1}{3!3!}+\frac{1}{2\cdot 5!}\right)$$
A third approach, let $M_{a,b}$ be the set of finite sequences $(a_0,\cdots,a_m)$ with $0\leq a_0<a$ and $a\leq a_i\leq b$ for $i>0.$ Let $M_{a,b,n}$ be the subset of such tuples with sum $n.$
Then $$F_{a,b}(n)=\sum_{(a_i)\in M_{a,b,n}}\binom{n}{a_0,a_1,\dots,a_m}$$ where $\binom{n}{a_0,\dots,a_m}$ is the multinomial.
When $a=3,b=6, n=7,$ $$M_{a,b,n}=\{(0,3,4),(0,4,3),(1,3,3),(1,6),(2,5)\}$$
So, $$\begin{align}F_{3,6}(7)&=\binom{7}{0,3,4}+\binom7{0,4,3}+\binom7{1,3,3}+\binom7{1,6}+\binom7{2,5}\\ &=2\binom73 + \frac{7!}{3!3!1!}+\binom71+\binom72 \end{align}$$