Let $u = (u_1, u_2)$ and $x=(x,y)$ a rotation of u by an angle θ. Then $\left\lVert u \right\rVert = \left\lVert x \right\rVert$
We know: $$cosθ = {{u \cdot x}\over \left\lVert u \right\rVert \cdot \left\lVert x \right\rVert} = {{u \cdot x}\over \left\lVert u \right\rVert \cdot \left\lVert u \right\rVert} = {{u \cdot x}\over \left\lVert u \right\rVert^2} = {{u \cdot x} \over u \cdot u} \Leftrightarrow (u \cdot u) cosθ = u \cdot x \tag 1 $$
Expanding $(1)$: $$(u_1^2 +u_2^2)cosθ = u_1x +u_2y \tag 2 $$
We also know: $$\left\lVert u \right\rVert = \left\lVert x \right\rVert \Leftrightarrow \sqrt{u_1^2 +u_2^2} = \sqrt{x^2 +y^2} \Rightarrow u_1^2 +u_2^2 = x^2 +y^2 \tag 3$$
For any set of $(u_1,u_2, θ)$ we now have a system of two equations and two unknowns. We should get two vectors that form an angle $θ$ with $u$ out of that.
But I was trying to combine these two formulas ($(2)$ and $(3)$) into a single generic one and I failed. Is it possible to do that?
Or could we solve $(1)$ for $x$ some way to get a vector equation for the rotation? $(x,y) = x_{something}\ cosθ \ (u_1, u_2) $ would be nice.
I know there is a formula for vector rotations using matrix transformations but I would like to know if it is possible to do the same thing with the aforementioned equations.
Suppose that $u_1,$ $u_2,$ and $\theta$ are given. First solve for $y$ in Equation $(2)$: $$ y = \frac1{u_2}(u_1^2 +u_2^2)\cos\theta - \frac{u_1}{u_2}x. \tag4$$
Now let $h = \frac1{u_2}(u_1^2 +u_2^2)\cos\theta$ and let $m = \frac{u_1}{u_2}.$ Note that for any given value of $u_1,$ $u_2,$ and $\theta,$ we can treat $h$ and $m$ as constants. Likewise we can set $r = \sqrt{u_1^2 + u_2^2}$ and treat $r$ as a constant. You don't really need to define these extra symbols, but they are a reminder of some important quantities that do not depend on $x$ or $y,$ and they let you write the equations a little quicker and therefore manipulate them a little easier.
Now we can write Equation $(4)$ as $$ y = h - mx \tag5$$ and Equation $(3)$ as $$ x^2 + y^2 = r^2 . \tag6$$
Use Equation $(5)$ to substitute for $y$ in Equation $(6)$: $$ x^2 + (h - mx)^2 = r^2. $$
Expand $(h - mx)^2$ and collect the terms in $x^2$ and $x$; also make a single constant term: $$ (m^2 + 1)x^2 - 2hmx + (h^2 - r^2) = 0. $$
This is a quadratic equation, $ax^2 + b + c=0$ with $a=m^2+1,$ $b = -2hm,$ and $c = h^2 - r^2.$ Solving it, \begin{align} x &= \frac{2hm \pm \sqrt{4h^2m^2 - 4(m^2 + 1)(h^2 - r^2)}}{2(m^2 + 1)}\\ &= \frac{2hm \pm \sqrt{4h^2m^2 - 4(h^2m^2 + h^2 - (m^2+1)r^2)}}{2(m^2 + 1)}\\ &= \frac{hm \pm \sqrt{(m^2+1)r^2 - h^2}}{m^2 + 1}. \end{align}
Now let's put back the original symbols, but carefully: \begin{align} h^2 &= \frac1{u_2^2}(u_1^2 +u_2^2)^2\cos^2\theta = \frac{r^4}{u_2^2}\cos^2\theta, \\ m^2 + 1 &= \frac{u_1^2}{u_2^2} + 1 = \frac{u_1^2 + u_2^2}{u_2^2} = \frac{r^2}{u_2^2},\\ (m^2+1)r^2 - h^2 &= \frac{r^4}{u_2^2} - \frac{r^4}{u_2^2}\cos^2\theta = \frac{r^4}{u_2^2} \sin^2\theta, \\ \sqrt{(m^2+1)r^2 - h^2} &= \frac{r^2}{u_2} \sin\theta, \\ hm &= \frac{u_1}{u_2^2}(u_1^2 +u_2^2)\cos\theta = \frac{u_1r^2}{u_2^2}\cos\theta. \end{align}
Therefore \begin{align} x &= \frac{\dfrac{u_1r^2}{u_2^2}\cos\theta \pm \dfrac{r^2}{u_2} \sin\theta} {\dfrac{r^2}{u_2^2}} \\ &= u_1 \cos\theta \pm u_2 \sin\theta, \\[1ex] y &= \frac{u_1^2 +u_2^2}{u_2}\cos\theta - \frac{u_1}{u_2}(u_1 \cos\theta \pm u_2 \sin\theta) \\ &= \frac1{u_2}((u_1^2 +u_2^2)\cos\theta - (u_1^2 \cos\theta \pm u_1u_2 \sin\theta)) \\ &= \frac1{u_2}(u_2^2\cos\theta \mp u_1u_2 \sin\theta) \\ &= u_2\cos\theta \mp u_1 \sin\theta, \\ \end{align} which is the same as you get from the rotation matrix. The $\pm$ and $\mp$ occur because your equations are not sufficient to distinguish a rotation by $\theta$ from a rotation by $-\theta,$ so we get solutions for two rotation matrices.