Alright, so I have been trying to get this substitution to work and I'm failing completely.
Maybe I'm missing something obvious or not thinking straight, I'm a bit sleep deprived.
Given this ODE
$x^{2}y^{''} - 5xy^{'} + 9(x^{6}-8)y = 0$
And these substitutions
$y = x^{3}u$
$z = x^{3}$
I'm supposed to reduce down to Bessel's formula and solve
Using the chain rule I get
$y^{'} = 3x^{2}u + x^{3}u^{'}$
$y^{''} = 6xu + 6x^{2}u^{'} + x^{3}u^{''}$
Substituting into given ODE I get
$x^{5}u^{''}+x^{4}u^{'}+9x^{9}u-81x^{3}u = 0$
Substituting $x = z^{1/3}$ I get
$z^{5/3}u^{''}+z^{4/3}u^{'}+9z^{3}u-81z$
I have no idea how this is supposed to get me any closer to actually reducing it. Am I missing something obvious? Making a mistake in the substitution?
Your first error is that you compute the derivatives of $y(x)=x^3u(x)$ where it should be $y(x)=x^3u(x^3)$.
The indicial polynomial is $$ r(r-1)-5r-72=(r-3)^2-81. $$ The leading monomials of a Frobenius power series solution are thus $x^{3\pm 9}$. To center the powers one needs to set $y(x)=x^3u(s)$
Try with a fully parametrized substitution \begin{align} y(x)&=x^au(bx^c),~~~ s=bx^c \\ xy'(x)&=ax^au(s)+bcx^{a+c}u'(s) \\ x(xy'(x))'&=a^2x^au(s)+bc(2a+c)x^{a+c}u'(s)+b^2c^2x^{a+2c}u''(s) \\ \hline x(xy'(x))'-6xy'(x)+9(x^6-8)y(x)&= [a^2-6a+9(x^6-8)]x^au(s) \\&~~~~+[bc(2a+c)-6bc]x^{a+c}u'(s) \\&~~~~+[b^2c^2]x^{a+2c}u''(s) \end{align} The aim is the form $s^2u''(s)+su'(s)+(s^2-n^2)u(s)=0$.
We need $c=3$ to get a quadratic power in the zero-order term. The reduced equation now reads $$ s^2u''(s)+\tfrac13[2a-3]su'(s)+[b^{-2}s^2-8+\tfrac19(a^2-6a)]u(s)=0 $$ With $a=3$ to get the first-order coefficient correct to $+1$, and $b=1$ to remove the coefficient of the square in the zero-order term, the final form is $$ s^2u''(s)+su'(s)+(s^2-9)u(s)=0,~~~y(x)=x^3u(x^3) $$ which is indeed what the hints suggested.