Let us consider the Fourier transform of $\mathrm{sinc}$ function. As I know it is equal to a rectangular function in frequency domain and I want to get it myself, I know there is a lot of material about this, but I want to learn it by myself. We have $\mathrm{sinc}$ function whhich is defined as $$ \mathrm{sinc}(\omega_0\,t) = \sin(\omega_0\,t)/(\omega_0\,t). $$ Its Fourier transform $$ \int_{\Bbb R} \sin(\omega_0\,t) \,e^{-i\,\omega\,t}/(\omega_0\,t)\,\mathrm dt $$ can be represented as $$ \int_{\Bbb R} \sin(\omega_0\,t)\,(\cos(\omega\,t) - i \,\sin(\omega\,t))/(\omega_0\,t)\,\mathrm dt. $$ Because we can distribute in brackets and consider that integral of difference is equal differences of integrals, we get $$ \int_{\Bbb R} \sin(\omega_0 \,t) \cos(\omega\,t)/(\omega_o\,t)\,\mathrm dt - \int_{\Bbb R} \sin(\omega_0\,t) \sin(\omega\,t)/(\omega_o\,t)\,\mathrm dt $$
but first product is zero, right? Because sine and cosine are orthogonal, so how could continue? Please help me.
In terms of deriving the Fourier Transform, I will make some use of techniques highlighted in http://www.claysturner.com/dsp/FTofSync.pdf
Let us start with your expression
$$\int_{-\infty}^{\infty}\frac{\sin(\omega_0t)\cos(\omega t)}{(\omega_ot)}dt-\int_{-\infty}^{\infty}\frac{j\sin(\omega_0t)\sin(\omega t)}{(\omega_ot)}dt$$
Examining the integrals, the term $\omega_0t$ in the denominator makes evaluating the integral more involved.
We can use some relevant trigonometric identities so that we can express $$\sin(\omega_0t)\cos(\omega t)=\frac{1}{2}[\sin((\omega+\omega_0)t)-\sin((\omega-\omega_0)t)]$$
$$\sin(\omega_0t)\sin(\omega t)=\frac{1}{2}[\cos((\omega-\omega_0)t)-\cos((\omega+\omega_0)t)]$$
To deal with the awkward $\omega_0t$ term, we can use the following identity to convert the single integral into a double integral that is far nicer to evaluate: $$\frac{1}{\omega_0t}=\int_0^{\infty}e^{-\omega_0ts}ds$$
Thus the integral to evaluate is $$\int_0^{\infty}\int_{-\infty}^{\infty}\sin(\omega_0t)\cos(\omega t)e^{-\omega_0tx}dtdx-j\int_0^{\infty}\int_{-\infty}^{\infty}\sin(\omega_0t)\sin(\omega t)e^{-\omega_0tx}dtdx$$
which expands to $$\frac{1}{2}\int_0^{\infty}\int_{-\infty}^{\infty}\sin((\omega+\omega_0)t)e^{-\omega_0tx}dtdx-\frac{1}{2}\int_0^{\infty}\int_{-\infty}^{\infty}\sin((\omega-\omega_0)t)e^{-\omega_0tx}dtdx\\-\frac{j}{2}\int_0^{\infty}\int_{-\infty}^{\infty}\cos((\omega-\omega_0)t)e^{-\omega_0tx}dtdx+\frac{j}{2}\int_0^{\infty}\int_{-\infty}^{\infty}\cos((\omega+\omega_0)t)e^{-\omega_0tx}dtdx$$
We can exploit Fubini's theorem to rewrite the integral as
$$\frac{1}{2}\int_{-\infty}^{\infty}\left[\int_0^{\infty}\sin((\omega+\omega_0)t)e^{-\omega_0xt}dt\right]dx-\frac{1}{2}\int_{-\infty}^{\infty}\left[\int_0^{\infty}\sin((\omega-\omega_0)t)e^{-\omega_0xt}dt\right]dx\\-\frac{j}{2}\int_{-\infty}^{\infty}\left[\int_0^{\infty}\cos((\omega-\omega_0)t)e^{-\omega_0xt}dt\right]dx+\frac{j}{2}\int_{-\infty}^{\infty}\left[\int_0^{\infty}\cos((\omega+\omega_0)t)e^{-\omega_0xt}dt\right]dx$$
We shall use the following integral identities to calculate the above integrals :- $$\int_0^{\infty}\sin(at)e^{-st}dt=\frac{a}{a^2+s^2}$$ $$\int_0^{\infty}\cos(at)e^{-st}dt=\frac{s}{a^2+s^2}$$
This results in $$\frac{1}{2}\int_{-\infty}^{\infty}\left[\frac{(\omega+\omega_0)}{(\omega+\omega_0)^2+\omega_0^2x^2}\right]dx-\frac{1}{2}\int_{-\infty}^{\infty}\left[\frac{(\omega-\omega_0)}{(\omega-\omega_0)^2+\omega_0^2x^2}\right]dx\\-\frac{j}{2}\int_{-\infty}^{\infty}\left[\frac{\omega_0x}{(\omega-\omega_0)^2+\omega_0^2x^2}\right]dx+\frac{j}{2}\int_{-\infty}^{\infty}\left[\frac{\omega_0x}{(\omega+\omega_0)^2+\omega_0^2x^2}\right]dx\text{ (Eq. 1)}$$
To evaluate the real component integrals in (Eq. $1$), we use the following result:- $$\int_{-\infty}^{\infty}\frac{a}{a^2+s^s}ds=\frac{|a|}{a}\int_{-\infty}^{\infty}\frac{1}{1+y^2}dy=sgn(a)\left[\arctan y\right]^{\infty}_{-\infty}=sgn(a)\pi$$
This leads to $$\int_{-\infty}^{\infty}\left[\frac{(\omega+\omega_0)}{(\omega+\omega_0)^2+\omega_0^2x^2}\right]dx=sgn(\omega+\omega_0)\pi$$ $$\int_{-\infty}^{\infty}\left[\frac{(\omega-\omega_0)}{(\omega-\omega_0)^2+\omega_0^2x^2}\right]dx=sgn(\omega-\omega_0)\pi$$
As regards the imaginary component integrals, note that the numerator is simply a constant times the derivative of the denominator, so we have
$$\int_{-\infty}^{\infty}\left[\frac{\omega_0x}{(\omega-\omega_0)^2+\omega_0^2x^2}\right]dx=\frac{1}{2\omega_0}\left[\ln\{(\omega-\omega_0)^2+\omega_0^2x^2\}\right]_{-\infty}^{\infty}$$
$$\int_{-\infty}^{\infty}\left[\frac{\omega_0x}{(\omega+\omega_0)^2+\omega_0^2x^2}\right]dx=\frac{1}{2\omega_0}\left[\ln\{(\omega+\omega_0)^2+\omega_0^2x^2\}\right]_{-\infty}^{\infty}$$
Combining the imaginary component integrals as per the integral in (Eq. $1$) we wish to evaluate and noting that they have opposite signs, we have $$-\int_{-\infty}^{\infty}\left[\frac{\omega_0x}{(\omega-\omega_0)^2+\omega_0^2x^2}\right]dx+\int_{-\infty}^{\infty}\left[\frac{\omega_0x}{(\omega+\omega_0)^2+\omega_0^2x^2}\right]dx\\=\lim_{x\rightarrow\infty}\frac{1}{2\omega_0}ln\left[\frac{(\omega+\omega_0)^2+\omega_0^2x^2}{(\omega-\omega_0)^2+\omega_0^2x^2}\right]+\lim_{x\rightarrow -\infty}\frac{1}{2\omega_0}ln\left[\frac{(\omega-\omega_0)^2+\omega_0^2x^2}{(\omega+\omega_0)^2+\omega_0^2x^2}\right]=\frac{2}{\omega_0}\ln(1)=0$$ Thus the imaginary terms in the integral cancel out, leading to the integral in (Eq. $1$) being a real result, as follows $$\frac{\pi}{2}\left[sgn(\omega+\omega_0)-sgn(\omega-\omega_0)\right]$$
Putting everything together we have: $$\int_{-\infty}^{\infty}\frac{\sin(\omega_0t)\cos(\omega t)}{(\omega_ot)}dt-\int_{-\infty}^{\infty}\frac{j\sin(\omega_0t)\sin(\omega t)}{(\omega_ot)}dt=\frac{\pi}{2}\left[sgn(\omega+\omega_0)-sgn(\omega-\omega_0)\right]$$
The result is a rectangular function that starts from frequency $-\omega_0$ and ends at frequency $\omega_0$.