Linear Differential equation: $$ \frac{dy}{dx} + y\cdot\arctan(x) = 5e^{\cos(x)} $$
From this I know that:
$$ P(x) = \arctan(x) \\ F(x) = 5e^{\cos(x)} $$
Now determining the integrating factor $u$:
$$ u = e^{\int{{\arctan{x}\,dx}}} = e^{-\frac{1}{2} \log{(x^2+1)} + x\arctan{x}} = e^{\log{(x^2+1)}^{-\frac{1}{2}}} \cdot e^{x\arctan{x}} = {(x^2+1)}^{-\frac{1}{2}} \cdot e^{x\arctan{x}} $$
Now in multiplying this by the original equation:
$$ (x^2+1)^{\frac{1}{2}} \cdot e^{x\arctan{x}} \cdot \frac{dy}{dx} + (x^2+1)^{-\frac{1}{2}} \cdot e^{x\arctan{x}} \cdot y \arctan{x} = (x^2+1)^{-\frac{1}{2}} \cdot e^{x\arctan{x}} \cdot 5e^{\cos(x)} \\ \frac{d((x^2+1)^{-\frac{1}{2}} \cdot e^{x\arctan{x}} y)}{dx} = (x^2+1)^{-\frac{1}{2}} \cdot e^{x\arctan{x}} \cdot 5e^{\cos{x}} \\ $$
Integrating both sides:
$$ (x^2+1)^{-\frac{1}{2}} \cdot e^{x\arctan{x}} y = \int (x^2+1)^{-\frac{1}{2}} \cdot e^{x\arctan{x}} \cdot 5e^{\cos{x}}dx $$
In which I am having problems with the integral. Where did I go wrong and how do I resolve the the problem?
The first question — "Where did I go wrong?" — is pretty easy to answer: your integrating factor $u$ is very wrong. Your mistake is that you can't "integrate" like that in the exponent: $$\int e^{f(x)}\,dx \color{red}{\neq} e^{\int f(x)\,dx}.$$ In other words, while it's true that $$\int \arctan x\,dx=-\frac{1}{2}\ln(x^2+1)+x\arctan x+C$$ (which, by the way, you typed incorrectly, as $\ln x^2+1\color{red}{\neq}\ln(x^2+1)$), that does NOT make the antiderivative of the exponent what you said it would be: $$\int e^{\arctan x}\,dx\color{red}{\neq}e^{-\frac{1}{2}\ln(x^2+1)+x\arctan x}+C.$$
What to do with it is a whole different question… Where did this question come from? Are you sure you have the correct statement of the equation? I didn't try to integrate it by hand, but Wolfram Mathematica gives a very ugly antiderivative of $\int e^{\arctan x}\,dx$.