I am trying to evaluate and simplify $\cos[\cos^{-1}(\frac{3}{5}) + \frac{\pi}{3}]$.
I am getting $\frac{11}{10}$ but the answer is $\frac{3-4\sqrt{3}}{10}$
My Process:
$\cos[\cos^{-1}(\frac{3}{5}) + \frac{\pi}{3}]$
$\cos[\cos^{-1}(\frac{3}{5})] + \cos(\frac{\pi}{3})$
$(\frac{2}{2}) \cdot \frac{3}{5} + \frac{1}{2} \cdot (\frac{5}{5})$
$\frac{6}{10} + \frac{5}{10}$
$\frac{11}{10}$
You cannot separate out the $\cos$ function as you have done in step two.
You can remember this identity.
$$\cos(A+B)=\cos(A)\cos(B)-\sin(A)\sin(B)$$
Here $\arccos(\frac{3}{5})$ is an angle ( i.e., approximately $53$ degrees)
Using thus result you should get the desired answer.