As a part of my task considering characteristic functions I have to compute $\int_{\mathbb R} e^{-itx}dx$
The result i get is $\frac{1}{-it}e^{-itx}|^{\infty}_{-\infty}$, but I don't really know where to go from here. Could anyone help, please?
(Wolfram says the integral does not converge - my starting problem was given characteristic function $\phi (t)=\frac{1}{4}+\frac{1}{4}e^{-it}+\frac{1}{2}\frac{3}{4-e^{2it}}$ find the the density of random variable X - so how to find it given the brute force method (integration) does not work?)
In fact the integral does not converge in the classical sense. However, we have that the characteristic function of a random variable with probability $p>0$ of taking on the value $x$ will have a term of $p e^{i t x}$. In particular the characteristic function of a random variable which has probability $p$ of taking on the value $0$ will have a term of $p$. A consequence of that is that in the sense of distribution theory, the integral you have written is $\delta_0$, that is, the Dirac delta centered at zero.
In your case, let us expand the last term with the geometric series:
$$\frac{1}{2} \frac{3}{4-e^{2 i t}} = \frac{3}{8} \frac{1}{1-e^{2 i t}/4} = \frac{3}{8} \sum_{n=0}^\infty \left ( \frac{e^{2 i t}}{4} \right )^n.$$
This means that your variable has probability $1/4$ of being $-1$, probability $1/4+3/8=5/8$ of being $0$, and probability $\frac{3 \cdot 4^{-n}}{8}$ of being $2n$ for each natural number $n$.
The fact that this variable is actually discrete should explain why the integral defining its density fails to converge: it is because its density doesn't exist as a function, but rather only as a distribution.