Request vetting everything, including trigonometry part. However, the main issue is the second approach used to find the rotated points.
This question arose in context of a question for rotation (extrinsic, i.e. of the point & not frame) in counter-clockwise direction by $30^0$ for a line $y = 5x+1$. It was easy to understand the rotation matrix given as (on pg. #8 of the text by William Ford, titled: Numerical linear algebra with applications):
$\begin{pmatrix} \cos(\frac{\pi}6) & -\sin(\frac{\pi}6) \\ \sin(\frac{\pi}6) & \cos(\frac{\pi}6) \end{pmatrix}$ $\begin{pmatrix} x_1 \\ y_1 \end{pmatrix}$, with parametric form adopted leading to $x_1 = x, y_1=5x+1$.
The book then takes two points on the line, given as $(0,1), (1,6)$ corresponding to $x=0,1$.
But, a thought arose regarding expressing the second matrix instead as $\begin{pmatrix} r\cos\alpha \\ r\sin\alpha \end{pmatrix}$, with $\alpha$ being the original slope of line & $r$ as magnitude of the vector.
But, $r$ (magnitude of vector) is changing at each real value of $x$; but $\alpha$ is constant, irrespective of $r$.
So, the available strategy reduces to either:
(i) taking a common parametric form for $x_1, y_1$ as in book, & then taking two points on the line, or
(ii) taking $x_1=r\cos\alpha, y_1= r\sin\alpha$ with computation of two points as say $(-1,-4), (1,6)$; giving slope $=\frac {10}2= 5$. Alternatively, using limits can find the limiting value as $\alpha = \arctan(5 + \frac 1x)= 5$.
This leads to $\cos\alpha =\frac{1}{\sqrt{\tan^2(\alpha)+1}}=
\frac1{\sqrt{6}}, \sin\alpha = \frac5{\sqrt{6}}$, for $0\le x\le \frac{\pi}{2}$, i.e. in the first quadrant (where $\arctan$ is taken only in the interval $(-\frac{\pi}2, \frac{\pi}2)$, but want positive value for both $\cos\alpha, \sin\alpha$).
So, taking a fixed value of $r$, i.e. for a given point $(0,1)$, $r=1$.
So, get the rotation matrix as : $\begin{pmatrix}
x_1 \\
y_1
\end{pmatrix}$=$\begin{pmatrix}
\cos(\frac{\pi}6) & -\sin(\frac{\pi}6) \\
\sin(\frac{\pi}6) & \cos(\frac{\pi}6)
\end{pmatrix}$$\begin{pmatrix}
\frac1{\sqrt{6}} \\
\frac5{\sqrt{6}}
\end{pmatrix} \implies \begin{pmatrix}
0.866 & -0.5 \\
0.5 & 0.866
\end{pmatrix}$$\begin{pmatrix}
\frac1{\sqrt{6}} \\
\frac5{\sqrt{6}}
\end{pmatrix}$
$\implies x_1 = \frac{0.866-2.5}{\sqrt{6}}, y_1 = \frac{5*0.866+0.5}{\sqrt{6}}
$, as rotated coordinates for a given point $(0,1)$.
But, as the rotation is for a line, so need compute two points' rotation to get the new equation of rotated line, as given below:
Taking second point, let: $(x_2,y_2)=(1,6)$, find the rotated coordinates as above by just modifying $r=\sqrt{37}$ for the new coordinates of the vector (with the same slope $\alpha$ as earlier):
$\begin{pmatrix}
0.866 & -0.5 \\
0.5 & 0.866
\end{pmatrix}\begin{pmatrix}
\frac{\sqrt{37}}{\sqrt{6}} \\
\frac{5*\sqrt{37}}{\sqrt{6}}
\end{pmatrix}\implies x_2 = x_1*\sqrt{37}, y_2 =y_1*\sqrt{37}$
But, this gives slope of the rotated line as : $\frac{y_2-y_1}{x_2-x_1}=\frac{y_1}{x_1}= \frac{5*0.866+0.5}{0.866-2.5} = -2.95593635= -71.309^0$, or $-1.2445801$ radians.
(i) If I am correct, then need just one point's coordinates to find the slope of the rotated line, rather than two.
(ii) But, am not sure about the correctness (or the way I have shown it correct) of the rotated angle, as found below:
The angle $\alpha$ has value $=\arctan(5)=78.69^0$, or $1.3734$ radians. Adding $\frac{\pi}6$ radians or $30^0$ (positive, as counter-clockwise rotation) yields $108.69^0$.
This angle lies in 2nd quadrant, so $(108.69^0-90^0)=18.69^0$. This angle taken clockwise from the $x$-axis in 2nd quadrant is $= -(90^0-18.69^0)= -71.31^0$.