I'll start with an example before my question so the approach becomes clearer:
To expand the function $$f(x) = \begin{cases}1,&0<x<1\\0,&\text{otherwise}\end{cases}$$ in a series of Legendre polynomials $P_n(x)$, first we find the coefficients $c_n$:
$$c_n=\frac{2n+1}{2}\int_{-1}^1 f(x)P_n(x)~dx = \frac{2n+1}{2}\int_0^1P_n(x)~dx$$
after applying Rodrigues' formula: $$c_n=\frac{2n+1}{2}\int_0^1\frac{1}{2^n n!}\frac{d^n}{dx^n}(x^2-1)^n~dx=\frac{2n+1}{2^{n+1}n!}\frac{d^{n-1}}{dx^{n-1}}(x^2-1)^n~\Biggr|_{0}^{1}$$
This terminates at $x=\pm1$, and performing a binomial expansion about $x=0$, we end up with: $$c_n=-\frac{2n+1}{2^{n+1}n!}\sum_{k=0}^{\infty}(-1)^{n-k}\binom{n}{k}\frac{d^{n-1}}{dx^{n-1}}x^{2k}$$
This is nonzero only for $2k=n-1$. Which means $n=2k+1$, thus only odd terms survive, and hence: $$c_n=-\frac{4k+3}{4^{k+1}(2k+1)!}(-1)^{k+1}\binom{2k+1}{k}(2k)!$$
Noting that $c_0=\frac{1}{2}$, the final result is: $$f(x)\sim\frac{1}{2}+\sum_{k=0}^{\infty}\frac{(-1)^k(4k+3)(2k)!}{4^{k+1}k!(k+1)!}P_{2k+1}(x)$$
Now here is my question: I understand (or at least grasp) the methodology here, But don't know how to apply it to a similar function with just one (not so) minor adjustment. If we consider the function: $$g(x) = \begin{cases}1,&0<x<a\\0,&\text{otherwise}\end{cases}\qquad(a<1)$$ Then there is no $x=1$ to terminate $(x^2-1)^n$, so the term at $x=a$ does not go away. I end up with a similar expression as in the example above but with two binomial expansions: one about $x=a$ minus the other at $x=0$. But I just get $n=2k+1$ for both series and they end up being equal since the $n-1$ derivatives of $(x-a)^{2k}$ differentiates it to $(2k)!$ so the result is just zero, which I don't believe it to be.
It was suggested to use the recursion formula: $$P'_{n+1}(x)-P'_{n-1}(x) = (2n+1)P_n(x).$$ But I can't get that to lead anywhere either, I expand both derivatives of $P$ in binomial series about $0$ and $a$, but just end up with the coefficients $c_n$ being zero. What is missing here?
Using the hint, isn't it just as simple as follows? \begin{align}c_n&=\frac{2n+1}{2}\int_{-1}^1 g(x)P_n(x)\,dx\\&=\frac12\int_0^a\big(P_{n+1}'(x)-P_{n-1}'(x)\big)\,dx\\&=\frac12\big(P_{n+1}(a)-P_{n-1}(a)-P_{n+1}(0)+P_{n-1}(0)\big)\end{align}