This problem can also be asked like this: "In how many different ways can we distribute $17$ balls to $5$ cells, while keeping the condition that in every cell the possible range of balls is $1 - 6$.
I managed to rewrite it to this particular form, but now stumped and can't seem to manage any progress.
Writing it in some sort of function form:$x_{1}+x_{2}+x_{3}+x_{4}+x_{5}=17$
and every $x_i$ is between $1$ and $6$.
answer: $\binom{16}{12} - 5\binom{10}{6} + \binom{5}{2} = 780$
We wish to find the number of solutions of the equation $$x_1 + x_2 + x_3 + x_4 + x_5 = 17 \tag{1}$$ in positive integers not larger than $6$.
A particular solution of equation 1 corresponds to the placement of four addition signs in the sixteen spaces between successive ones in a row of seventeen ones.
$$1 \square 1 \square 1 \square 1 \square 1 \square 1 \square 1 \square 1 \square 1 \square 1 \square 1 \square 1 \square 1 \square 1 \square 1 \square 1 \square 1$$ For instance, placing addition signs in the fourth, sixth, eleventh, and fifteenth spaces yields $$1 1 1 1 + 1 1 + 1 1 1 1 1 + 1 1 1 1 + 1 1$$ which corresponds to the solution $x_1 = 4$, $x_2 = 2$, $x_3 = 5$, $x_4 = 4$, $x_5 = 2$. Thus, the number of solutions of equation 1 is $$\binom{16}{4}$$ since we must select which four of the sixteen spaces between successive ones in a row of seventeen ones will be filled with addition signs.
A particular solution of the equation $$x_1 + x_2 + x_3 + \cdots + x_n = k$$ in the positive integers corresponds to the placement of $n - 1$ addition signs in the $k - 1$ spaces between successive ones in a row of $k$ ones. The number of such solutions is $$\binom{k - 1}{n - 1}$$ since we must select which $n - 1$ of the $k - 1$ spaces between successive ones in a row of $k$ ones will be filled with an addition sign.
However, the $\binom{16}{4}$ solutions of equation 1 include those in which one or more of the variables exceeds $6$. We must subtract these from the total. Notice that at most two of the variables can exceed $6$ since $2 \cdot 7 + 3 \cdot 1 = 17$.
Suppose a variable exceeds $6$. There are five ways to choose this variable. Suppose it is $x_1$. Then $x_1' = x_1 - 6$ is a positive integer. Substituting $x_1' + 6$ for $x_1$ in equation 1 yields \begin{align*} x_1' + 6 + x_2 + x_3 + x_4 + x_5 & = 17\\ x_1' + x_2 + x_3 + x_4 + x_5 & = 11 \tag{2} \end{align*} Equation 2 is an equation in the positive integers with $$\binom{11 - 1}{5 - 1} = \binom{10}{4}$$ solutions. By symmetry, there are an equal number of solutions for each variable that exceeds $6$. Hence, there are $$\binom{5}{1}\binom{10}{4}$$ solutions in which one of the variables exceeds $6$.
However, if we subtract the number of solutions in which one of the variables exceeds six from the total, we will have subtracted too much since we will have subtracted each case in which two of the variables exceeds six twice, once for each way we could designate one of the variables as the variable that exceeds $6$. Since we only want to subtract those cases once, we must add them back.
There are $\binom{5}{2}$ ways to select two variables which each exceed $6$. Suppose they are $x_1$ and $x_2$. Let $x_1' = x_1 - 6$; let $x_2' = x_2 - 6$. Then $x_1'$ and $x_2'$ are positive integers. Substituting $x_1' + 6$ for $x_1$ and $x_2' + 6$ for $x_2$ in equation 1 yields \begin{align*} x_1' + 6 + x_2' + 6 + x_3 + x_4 + x_5 & = 17\\ x_1' + x_2' + x_3 + x_4 + x_5 & = 5 \tag{3} \end{align*} Equation 3 is an equation in the positive integers with $$\binom{5 - 1}{5 - 1} = \binom{4}{4} = 1$$ solution. By symmetry, there are an equal number of solutions for each pair of variables that exceed $6$. Hence, there are $$\binom{5}{2}\binom{4}{4}$$ solutions in which two of the variables exceed $6$.
By the Inclusion-Exclusion Principle, the number of solutions of equation 1 in positive integers not larger than $6$ is $$\binom{16}{4} - \binom{5}{1}\binom{10}{4} + \binom{5}{2}\binom{4}{4}$$