I tried to solve the integral below using integration by parts $$\int_0^t\cos(x)\cos(t-x)dx=\frac{1}{2}(\sin(t)+t\cos(t))$$
It seemed solvable through doing integration by parts twice, but it hasn't worked for me yet... tcos(t) doesn't come up!
I know how it can be solved using properties of trig function, why can't it be solved by integration by parts?
You can do it by integration by parts, but you have to start off in a slightly funny direction to stop the final integral cancelling with the original: $$ I = \int_0^t 1 \cdot \cos{x}\cos{(t-x)} \, dx = [x\cos{x}\cos{(t-x)}]_0^t - \int_0^t x(-\sin{x}\cos{(t-x)}+\cos{x}\sin{(t-x)}) \, dx \\ = t\cos{t} + \int_0^t x(\sin{x}\cos{(t-x)}-\cos{x}\sin{(t-x)}) \, dx, $$ giving something that looks like one of the terms. Now integrate the first term in the integral by parts, $$ \int_0^t x\sin{x}\cdot \cos{(t-x)} \, dx = [-x\sin{x}\sin{(t-x)}]_0^t + \int_0^t (x\cos{x}-\sin{x}) \sin{(t-x)} \, dx. $$ The first term in the remaining integral cancels with the second term in the integral from the first integration by parts, so $$ \int_0^t \cos{x}\cos{(t-x)} \, dx = t\cos{t} - \int_0^t \sin{x}\sin{(t-x)} \, dx. $$ To deal with the remaining integral we integrate by parts again: $$ - \int_0^t \sin{x}\sin{(t-x)} \, dx = [\cos{x}\sin{(t-x)}]_0^t-\int_0^t \cos{x}\cos{(t-x)} \, dx \\ = \sin{t}-I, $$ so $$ I = t\cos{t}+\sin{t}-I, $$ or $$ I = \frac{1}{2}(t\cos{t}+\sin{t}), $$ as expected.
It was not at all obvious that this would work. In general, $$ \int fg' \, dx = xfg' - \int x(f'g'+fg'') \, dx \\ = x(fg' - f'g) + \int [(xf''+f')g-xfg''] \, dx \\ = x(fg'-f'g) + fg-\int fg' \, dx + \int x(f''g-fg'') \, dx, $$ so $$ \int fg' \, dx = \frac{1}{2}(x(fg'-f'g)+fg)+\frac{1}{2}\int x(f''g-fg'') \, dx, $$ and it so happens that in this case the integral on the right vanishes (as it will for any two solutions to a differential equation of the form $y''+q(x)y=0$, since $f''g-fg''$ is the derivative of the Wronskian, and this vanishes since the coefficient of $y'$ is zero).
A way to do this without a strange integration by parts is to look at $$ \lim_{a \to 1} \int_0^t \cos{ax}\cos{(t-x)} \, dx: $$ the integral and the limit can be interchanged because everything is continuous and the limit function is continuous, but $\int_0^t \cos{ax}\cos{(t-x)} \, dx$ can be done by parts in the usual way: we find $$ \int_0^t \cos{ax}\cos{(t-x)} \, dx = \dotsb = \sin{t}-a\sin{t} + \int_0^t \cos{ax}\cos{(t-x)} \, dx, $$ so $$ \int_0^t \cos{ax}\cos{(t-x)} \, dx = \frac{\sin{t}+a\sin{at}}{1-a^2}, $$ and taking the limit gives the result (although will require some trig identities).