I am trying to use this definition of the binomial coefficient:
$$\binom \alpha k = \frac{\alpha^{\underline k}}{k!} = \frac{\alpha(\alpha-1)(\alpha-2)\cdots(\alpha-[k-1])}{k(k-1)(k-2)\cdots 1}$$
To check that ${-n \choose -n}=0.$ Using it for a positive $k$, I'll stop
$$a(a-1)(a-2)\dots (a-[k-1])$$
When I reach $[k-1]$ but I don't know where is this stopping point when $n$ is negative. I know that I could do the following:
$${-n \choose -n}=\frac{(-n)!}{(-n)!([-n]-[-n])}=\frac{(-n)!}{(-n)!\cdot 0}=undefined$$
But this doesn't really solve my problem. My other trial was to do this:
$$\frac{(-n)(-n-1)(-n-2)\dots(-n-[-n-1])}{(-n)(-n-1)(-n-2)\dots(-n-[-n-1])}$$
But it also doesn't seem too revealing.
Your definition of the binomial coefficient is incomplete: it should read
$$\binom{x}k=\begin{cases} \frac{x^{\underline k}}{k!},&\text{if }k\ge 0\\\\ 0,&\text{if }k<0\;, \end{cases}$$
where $k$ is in all cases required to be an integer. Thus, if $n$ is a positive integer we have $\binom{-n}{-n}=0$ by definition.