(a) Find the exponential generating function (in closed form) for the number of ways to place n students into 3 classes (one MATH, one BIOLOGY and one PHYSICS), such that the MATH class does not have exactly one student. (Classes are allowed to be empty).
(b) Using the e.g.f. found in part (a). Find the formula for the coefficient of $\frac{x^n}{n!}$.
Workings:
$f(x) = (1 + \frac{x^2}{2!} + \frac{x^3}{3!} + ...) (1 + x + \frac{x^2}{2!} + ...) (1 + x + \frac{x^2}{2!} + ...)$
$f(x) = (e^x - x)e^xe^x$
$f(x) = (e^x-x)e^{2x}$
b.
$f(x) = (e^x-x)e^{2x}$
$f(x) = e^{3x}-xe^{2x}$
$e^{3x}-xe^{2x} = \sum\limits_{n=0}^{\infty}3^n\frac{x^n}{n!}-x\sum\limits_{r=0}^{\infty}2^n\frac{x^n}{n!}$
The coefficient of $\frac{x^n}{n!}$ is $3^n - x*2^n$.
I'm wondering if I did this correctly.
Any help will be appreciated.
You were doing fine until you tried to extract the coefficient of $\frac{x^n}{n!}$ from $xe^{2x}$. The coefficient is just that: it cannot involve $x$. Thus, you need to incorporate that extra factor of $x$ in the summation:
$$x\sum_{n\ge 0}2^n\frac{x^n}{n!}=\sum_{n\ge 0}2^n\frac{x^{n+1}}{n!}\;.$$
The problem now is that we have $\frac{x^{n+1}}{n!}$ and not $\frac{x^n}{n!}$. As a first step, notice that
$$\frac{x^{n+1}}{n!}=(n+1)\frac{x^{n+1}}{(n+1)!}\;,$$
so we can rewrite the summation as
$$\sum_{n\ge 0}2^n(n+1)\frac{x^{n+1}}{(n+1)!}$$
and do a simple index shift to get
$$\sum_{n\ge 1}n2^{n-1}\frac{x^n}{n!}\;.$$
And since $n2^{n-1}=0$ when $n=0$, we can start the summation at $n=0$ without doing any harm. Thus,
$$e^{3x}-xe^{2x}=\sum_{n\ge 0}3^n\frac{x^n}{n!}-\sum_{n\ge 0}n2^{n-1}\frac{x^n}{n!}=\sum_{n\ge 0}\left(3^n-n2^{n-1}\right)\frac{x^n}{n!}\;,$$
and the coefficient of $\frac{x^n}{n!}$ is $3^n-n2^{n-1}$.
Added: Note that you can do some checking of your answer. By brute force calculation we have
$$\begin{align*} f(x)&=\left(1+\frac{x^2}{2!}+\frac{x^3}{3!}+\ldots\right)e^{2x}\\ &=\left(1+\frac{x^2}{2!}+\frac{x^3}{3!}+\ldots\right)\left(1+2x+\frac{4x^2}{2!}+\frac{8x^3}{3!}+\ldots\right)\\ &=1+2x+5\frac{x^2}{2!}+15\frac{x^3}{3!}+49\frac{x^4}{4!}+\ldots\;, \end{align*}$$
and the coefficients $1,2,5,15$, and $49$ are indeed those given by the formula.