I have a frequency domain graph as shown. I need to "sharpen" the curve to get a better response, and computing large butterworth orders is not possible on my machine. Hence, I would like to know if there is a math technique to sort of sharpen the curve below.
It looks like you have two transfer functions here. Call the blue $H_1(f)$ and the green $H_2(f)$. Presumably, the red is the product $H_1(f)H_2(f)$.
It seems that the important behavior is that near $f = 1000\,Hz$. We can "sharpen" it, perhaps, in the following way:
Instead of $H_1(f)$ and $H_2(f)$, use
$H_1(a f - (a-1)(1000\,Hz))$ and $H_2(a f - (a-1)(1000\,Hz))$
for some choice of $a>1$
It should be easy to see which change to the original function causes this using the properties of the inverse Fourier transform.