Basically, I want to show the limit of the second function over the first function as $n$ approaches infinity is $0$, but I'm not sure how to get past the complicated nature of the second function. I would try to show that $3^{n^2}$ grows faster than $2^{n^2}$, which is faster than the second function, but the limit of those two functions is $\frac{2}{3}$ I think.
Also, the logarithms are base $2$, if that changes anything.
Well, you have that \begin{align} \frac{3^{n^2}}{n^8} = \exp\left( n^2\log 3\right)\cdot n^{-8} = \exp(n^2\log 3)\cdot \exp(-8\log n) = \exp(\log(3) n^2-8\log n) \end{align} versus \begin{align} \exp(\log(2)n\log(n) \log\log(n) ). \end{align} Since $n^2$ grows faster than $n\log n \log\log n$, then we see that $\frac{3^{n^2}}{n^8}$ grows faster than $2^{n\log n \log\log n}$.