I'm trying to calculate the expansion of $f : [0,1]\to\mathbb{R}$ given by $f(x)=x^2$ in a Fourier-Bessel series of zeroth order. In that case let $J_0$ be the $0$-th order Bessel function and $\alpha_n$ its $n$-th zero. In that case the expansion is
$$f(x)=\sum_{n=0}^\infty c_n J_{0,n}(x)$$
with $J_{0,n}(x)=J_0(\alpha_n x)$. Considering then the inner product $\langle,\rangle$ given by
$$\langle f,g\rangle = \int_0^1 f(x)g(x)xdx$$
we have the orthogonality relation for $J_{0,n}$ and $J_{0,m}$
$$\langle J_{0,n},J_{0,m}\rangle = \dfrac{1}{2}J_1^2(\alpha_n)\delta_{nm}.$$
In that case the coefficients $c_n$ are given by
$$c_n = \dfrac{\langle f,J_{0,n}\rangle}{\langle J_{0,n},J_{0,n}\rangle}=\dfrac{2}{J_1^2(\alpha_n)}\int_0^1 f(x)J_0(\alpha_n x)xdx.$$
To compute this integral for $f(x)=x^2$ I did the following: first I've changed variables in the integral, using $u=\alpha_n x$ and $du=\alpha_n dx$ so that
$$\int_0^1 f(x)J_0(\alpha_n x)xdx = \int_0^{1}x^3J_0(\alpha_n x)dx = \dfrac{1}{\alpha_n^4}\int_0^{\alpha_n} u^3 J_0(u)du.$$
After that I've used the definition of $J_0$:
$$J_0(x) = \sum_{k=0}^\infty \dfrac{(-1)^k}{\Gamma(k+1)\Gamma(k+1)}\left(\dfrac{x}{2}\right)^{2k},$$
so that we have
$$\langle f,J_{0,n}\rangle =\dfrac{1}{\alpha_n^4}\int_0^{\alpha_n}\sum_{k=0}^\infty \dfrac{(-1)^k}{\Gamma(k+1)\Gamma(k+1)}\dfrac{u^{2k+3}}{2^{2k}}du$$
$$\langle f,J_{0,n}\rangle=\dfrac{1}{\alpha_n^4}\sum_{k=0}^\infty\dfrac{(-1)^k}{\Gamma(k+1)\Gamma(k+1)}\dfrac{\alpha_n^{2k+4}}{2^{2k}(2k+4)}.$$
I know the answer is
$$c_n = \dfrac{2(\alpha_n^2-4)}{\alpha_n^3 J_1(\alpha_n)}$$
but I'm stuck there. How can I proceed this? Is there anything I did wrong? Any help is appreciated.
I suggest that you instead try to integrate using the differential equation $$ x^2J_\nu''(x)+xJ_\nu'(x)+(x^2-\nu^2)J_\nu(x)=0 $$ and the recurrence formulas $$ \frac{2\nu}{x}J_\nu(x)=J_{\nu-1}(x)+J_{\nu+1}(x) \quad \text{and} \quad 2J'_\nu(x)=J_{\nu-1}(x)-J_{\nu+1}(x). $$ I show you a first step, and you tell me if you need further steps.
Using the differential equation, $$ \int x^3 J_0(x)\,dx=-\int x^3 J''_0(x)+x^2 J'_0(x)\,dx. $$ Integrating the first term by parts, $$ \int -x^3 J_0''(x)-x^2 J_0'(x)\,dx=-x^3J_0'(x)+2\int x^2 J_0'(x)\,dx $$ Replacing $J_0'(x)=-J_1(x)$, we get $$ \int x^3 J_0(x)\,dx=x^3J_1(x)+2\int x^2 J_0'(x)\,dx $$ Working with the second integral, you should in the end get $$ \int x^3 J_0(x)\,dx = (x^3 -4x)J_1(x)+2x^2 J_0(x)+C. $$ It is a good exercise which I leave to you.