I have some questions about the solution to the five pills problem which is part of the list of exercises based on Feynman's lectures. 
In the solution above m is the number of types of pills and there are n pills of each type for a total of mn pills. K is the number of pills drawn without replacement.
I understand how the principle of inclusion exclusion was used to find L(m,n,k) but am unsure how in the above solution the expression for O(m,n,k) was arrived at. Also, how can it be valid if in the summation j ranges from 0 to m and j-m-1 is negative for these values of j?
There are altogether $\binom{mn}k$ combinations of $k$ pills, and $L(m,n,k)$ of them lack at least one pill type, so $\binom{mn}k-L(m,n,k)$ of them contain all types. And
$$\begin{align*} \binom{mn}k&-L(m,n,k)\\ &=\binom{mn}k-\sum_{j=1}^m(-1)^{j-1}\binom{m}j\binom{mn-jn}k\\ &=(-1)^0\binom{m}0\binom{mn-0\cdot n}k-\sum_{j=1}^m(-1)^{j-1}\binom{m}j\binom{mn-jn}k\\ &=(-1)^0\binom{m}0\binom{mn-0\cdot n}k+\sum_{j=1}^m(-1)^j\binom{m}j\binom{mn-jn}k\\ &=\sum_{j=0}^m(-1)^j\binom{m}j\binom{(m-j)n}k\\ &=\sum_{j=0}^m\binom{j-m-1}j\binom{(m-j)n}k \end{align*}$$
by virtue of the identity
$$\binom{r}k=(-1)^k\binom{k-r-1}k$$
(with the factor of $(-1)^k$ brought over to the other side).
The binomial coefficient $\binom{r}k$ is defined for all real (and even complex) numbers $r$ and integers $k$ as follows:
$$\binom{r}k=\begin{cases} \frac{r^{\underline{k}}}{k!},&\text{if }k\ge 0\\ 0,&\text{if }k<0\,, \end{cases}$$
where
$$r^{\underline{k}}=r(r-1)\ldots(r-k+1)=\prod_{i=0}^{k-1}(r-i)$$
is a falling factorial. Thus, negative values of $j-m-1$ are not a problem. In fact you can see that $(j-m-1)^{\underline{j}}$ is a product of $j$ negative numbers and so is negative when $j$ is odd and positive when $j$ is even, just like $(-1)^j$.