I want to solve the recurrence $T(n) = 64 T(n/4) + n^{\log n}$
From the master theorem we have that $f(n)=n^{\log n}$ and I want to compare this to $n^{\log_ba} = n^3$. Case 1 of the master theorem states that if $f(n) = O(n^{3-\epsilon})$ then $T(n) = O(n^3)$. I am trying to figure out if this case doesn't hold.
Since $\lim_{n\to\infty} \log n = \infty$ it seems that $n^{\log n}= \Omega(n^{3+e})$ even though this happens very slowly?
$\log n$ is eventually greater than $3 + \epsilon$ for sufficiently large $n$ so $n^{\log n} \geq n^{3 + \epsilon}$ for sufficiently large $n$. Hence $n^{\log n} = \Omega(n^{3 + \epsilon})$. Your reasoning is correct.
The master theorem doesn't tell you much here since by similar reasoning $n^{\log n} = \Omega(n^c)$ for any $c > 0$ so work to split/recombine a problem greatly dominates subproblems.