I have two points $(x_1,y_1)$ and $(x_2,y_2)$. Using dot product I have calculated the angle between the two. Let's call this angle $A$. Now, I want to rotate $(x_2,y2)$ around $(x1,y1)$ such that resulting angle between the $(x_1,y_1)$ and $(x'_2, y'_2)$ becomes $5^\circ$. So, I continue like below. Find angle of rotation which will $A' =(A -5)$. So new points will be
$$x'_2 = \cos(A')(x_2 - x_1) - \sin(A') (y_2 - y_1) + x_1$$ $$y'_2 = \sin(A')(x_2 - x_1) + \cos(A') (y_2 - y_1) + y_1$$
Is this correct?
if it's correct then the problem is that if I recalculate the angle using dot product between $(x_1,y_1)$ and $(x'_2,y'_2)$ then the resulting angle is not $5^\circ$. How can his be. My understanding of rotation and resulting new angle is wrong?
Using the dot product you can calculate the angle between two vectors, not between two points (What is the angle'' between two points''?).
So, if I well understand your problem, you have two vectors: $\vec p_1=(x_1.y_1)^T$ and $\vec p_2=(x_2.y_2)^T$ and you want a new vector $\vec p=(x,y)$ such that.
$$ \begin{cases} \frac{\vec p_1 \cdot \vec p}{|\vec p_1| |\vec p|}=\cos 5°\\ |\vec p -\vec p_1|=|\vec p_2-\vec p_1| \end{cases} $$ this is a system in the two unknowns $ (x,y)$ and solvin it you find the vector $\vec p$.
Now you can find the rotation angle $\theta$ of $\vec p_1$ around $p_1$ that gives $R_\theta(\vec p_2)=\vec p$ as the angle such that:
$$ \cos \theta =\frac{(\vec p-\vec p_1)\cdot(\vec p_2-\vec p_1)}{|\vec p-\vec p_1|^2} $$