I am trying to find a set of optimum parameters of my model that provide the best fit. I chose to make this into an optimization problem where I minimize my $\chi^2$ per degree of freedom and try to find the parameters for the minimum $\chi^2$. The dependence of $\chi^2$ on one of my parameters looks like the following:
As you can see, the dependence is very very noisy and the minimum presents as a very narrow valley. To find this minimum, I tried gradient descent only to realize that because the valley is very narrow, my gradient is going to be very large and even with a tiny learning rate, I move significantly away from my initial point thereby failing to work.
Then I thought maybe Newton's method would do but again realized that it would suffer the same issue. The first derivative, if we're already very close to the valley would be large and the second derivative would very likely be 0 or very close to 0, thereby failing again. My code moves from the starting point to a far away point just like in gradient descent.
So I was wondering if there is a method that might work for my case. I do know the starting point pretty well and right now I'm brute-forcing my way through it, where I choose a large number of points around my starting point and calculate $\chi^2$ for all of them and find the minimum but that is not very efficient.