I'm aware that this sequence converges to 0, but I'd like to know why my first instinct was wrong. I looked at this and took a cue from the old 1^inf technique of raising e to the natural log of the sequence:
$$\lim_{n\to \infty} n^{-n^2} \implies e^{\ln(n^{-n^2})}$$
leave e aside for now and concentrate on the natural log problem
$$-n^2\cdot \ln n\implies \frac{\ln n}{-n^{-2}}$$
by l'Hôpital's rule
$$\frac{\frac1n}{2n^{-3}} =\frac{n^2}2 $$
back to the original problem
$$\lim_{n\to \infty} e^{n^2/2} =( e^\infty)= \infty$$
The book offers a squeeze/sandwich theorem explanation, but I need to know what my mistake is so that I do not repeat it. Where did I go wrong here?
Note that
$$n^{-n^2}=e^{-n^2\log n}=\frac{1}{e^{n^2\log n}}\to 0$$
indeed
$$n^2\log n\to\infty$$