If $u(x)-\sin(x)=2\int_{0}^x \cos(x-t)u(t)dt$ then $u(x)$ equals
$(1)\frac{e^x}{x}$
$(2)\frac{x}{e^x}$
$(3)xe^x$
$(4)\frac{1}{xe^x}$
we have, $$u(x)-\sin(x)=2\int_{0}^x \cos(x-t)u(t)dt\tag{A}$$
Using Leibnitz's rule of differentiation under integral sign,we get $$u'(x)-\cos(x)=2[-\int_{0}^x \sin(x-t)u(t)dt+u(x)]$$
$$\implies u'(x)-u(x)-\cos(x)=-2\int_{0}^x \sin(x-t)u(t)dt$$Using Leibnitz's rule of differentiation under integral sign,we get
$$u''(x)-u'(x)+\sin(x)=-2\int_{0}^x \cos(x-t)u(t)dt\tag{B}$$ Using equations $(A)$ and $(B)$,we get $$u''(x)-u'(x)+\sin(x)=\sin(x)-u(x)$$
$$u''(x)-u'(x)+u(x)=0\tag{C}$$
Auxiliary equation for $(C)$ is $m^2-m+1=0\implies m=\frac{1\pm \iota\sqrt 3}{2}$
So,$u(x)=c_1\cos(\frac{\sqrt 3}{2})x+c_2\sin(\frac{\sqrt 3}{2})$,where $c_1,c_2$ are arbitrary constants.
which is not in any of the option,what is wrong with my solution?
You made a mistake here $$u'(x)-\cos(x)=2\left[-\int_{0}^x \sin(x-t)u(t)dt+u(x)\right]$$ Then after this you forgot the factor 2 this complicated the DE and the answer $$\implies u'(x)-\color{red}{2u(x)}-\cos(x)=-2\int_{0}^x \sin(x-t)u(t)dt$$ And use this to eliminate the constants: $$u(0)-\sin(0)=2\int_{0}^0 \cos(x-t)u(t)dt=0$$ $$ \implies u(0)=0$$ Do the same for $u'(0)$. $$u'(0)-\color{red}{2u(0)}-\cos(0)=-2\int_{0}^0 \sin(x-t)u(t)dt$$ $$ \implies u'(0)=1$$
Edit:
Note that since $u(0)=0 $ then answer 1 and 4 are wrong.