I have one non-linear function that define
$$E_x(a,b)=\int K_\sigma(y-x) \cdot(b-b. e^{-a\cdot f(y)} \,) dy$$
where $y$ is neighboor points of $x$; $f(y)$ is a function of $y$; and $a$ is constant.
I want to find a,b to minmize E(a,b). Hence, I take want derivative of this function with respect to $a$ and $b$. For derivative with respect to a, Emanuele Paolini helps me to do it:
$$
\frac{\partial}{\partial a}E_x(a,b)=\int b.f(y).K_\sigma(y-x)\cdot e^{-a\cdot f(y)}dy
=0 $$
And similar with b
$$
\frac{\partial}{\partial b}E_x(a,b)=\int K_\sigma(y-x)\cdot(1- e^{-a\cdot f(y)})dy=0
$$
Is my solution correct? From two above functions, How to find $a$ and $b$? Thank you so much
Your $E_x$ is linear in $b$, that helps... but the resulting system of equations is nasty. Except in very special cases of $K_\sigma$ and $f$ I'd say an analytical solution is hopeless. You should look for numerical methods, either for finding the minimum of $E_x$ directly or to solve the system of equations resulting from equating the derivatives to zero.
Unless you have reasonably simple formulas for the integral(s), you'll need to compute them by numerical methods, in which case you should be looking for methods that do not use derivatives to find the minimum/solve the system.