I have a least square problem: $$ \min \sum_j^p| f(x_j)-y_j |^2 \\ \text{where } f(x) = a x - \sum_{i}^{n} b_i J_1(c_i x) $$ where $J_1$ is the first order Bessel function. I have to find a set consists of $(2n+1)$ parameters $\{ a, b_1, ... b_n, c_1, ..., c_n \}$ that minimize the residual function.
Because the Bessel function oscillates all the time, it seems there are many possible ways to fit. And in most cases, the fitting curve also oscillates rapidly (see fig below), which looks like over-fitting.
I have used the L2 regularization method, but the regularization parameters determined by GCV method cannot work well (either too big that the fitting curve becomes a horizontal straight line, or too small that the curve still over-fitting). Is there some other ways to determined the regularization parameter?
Moreover, if I want to let the adjacent $b_i$ have the same sign, is there a possible way to add a punishment term? And how to determine the punishment strength?