Let's say I have a vector $\mathbf{A}$ which is represented in some coordinate system with unit basis vectors $\hat{\mathbf{e}}_\alpha$ and $\hat{\mathbf{e}}_\beta$, and has components along the $\alpha$ and $\beta$ directions of $u_\alpha$ and $u_\beta$ (which are known), respectively, i.e. \begin{equation} \mathbf{A} = u_\alpha\, \hat{\mathbf{e}}_\alpha + u_\beta\, \hat{\mathbf{e}}_\beta. \end{equation}
Now I have a second coordinate system with unit basis vectors $\hat{\mathbf{e}}_x$ and $\hat{\mathbf{e}}_y$, which are related to the previous ones through some rotation matrix $\mathbf{R}$, such that \begin{equation} \hat{\mathbf{e}}_\alpha = \mathbf{R}\,\hat{\mathbf{e}}_x \\ \hat{\mathbf{e}}_\beta = \mathbf{R}\,\hat{\mathbf{e}}_y \end{equation} In this basis, the vector $\mathbf{A}$ has components $u_x$ and $u_y$, such that \begin{equation} \mathbf{A} = u_x\, \hat{\mathbf{e}}_x + u_y\, \hat{\mathbf{e}}_y. \end{equation}
If $u_\alpha$ and $u_\beta$ are known, what is the correct notation to obtain $u_x$ and $u_y$ ? Intuitively, I would say that we need to project the vector $\mathbf{A}$ onto the second coordinate system (x,y), with the dot product, but how to correctly write this?
Is it correct to substitute the basis vector and write the following? \begin{align} \mathbf{A} &= u_\alpha\, \hat{\mathbf{e}}_\alpha + u_\beta\, \hat{\mathbf{e}}_\beta \\&= u_\alpha\, \mathbf{R}\,\hat{\mathbf{e}}_x + u_\beta\, \mathbf{R}\,\hat{\mathbf{e}}_y \end{align}
If so, then this looks like \begin{align} \mathbf{A} &= \begin{pmatrix} u_x \\ u_y \end{pmatrix} = \begin{pmatrix} u_\alpha\,\mathbf{R} \\ u_\beta\,\mathbf{R} \end{pmatrix} \end{align} ...which looks wrong. Where am I going wrong with notation here?
You're right to think of using $R$ to express $\hat{e}_\alpha = R\hat{e}_x$. It would go something like: $$\begin{align*}\mathbf{A} &= u_\alpha\hat{\mathbf{e}}_\alpha+u_\beta\hat{\mathbf{e}}_\beta \\ &= u_\alpha \mathbf{R}\hat{\mathbf{e}}_x+u_\beta\mathbf{R}\hat{\mathbf{e}}_y \\ &= \mathbf{R} (u_\alpha\hat{\mathbf{e}}_x+u_\beta\hat{\mathbf{e}}_y)\end{align*}$$
so you apply the rotation matrix (expressed in the x-y basis) to the vector (as represented in the rotated basis). Expressed vectorially:
$$\begin{pmatrix} u_x \\ u_y \end{pmatrix} = [\mathbf{R}]_{xy} \begin{pmatrix} u_\alpha \\ u_\beta \end{pmatrix}$$
where here I'm using the $[\cdot]$ notation to stress that this only works when $\mathbf{R}$ is expressed in the (x-y basis). If it is expressed in the rotated basis, a bit more work is necessary, but given that you seem to define the rotated basis in terms of the rotation acting on the x-y basis I'm assuming that's not the case.
For more details, you might want to check out some resources on change of basis. 3Blue1Brown's Essence of Linear Algebra comes to mind, obviously, but there are lots of other choices out there (even Wikipedia usually does a pretty good job).
Edit:
The reason you can't write $\mathbf{R}$ in the vector itself is that each entry of the vector has to be a scalar (single number), whereas $\mathbf{R}$ is a matrix / operator.
Here's an explanation that's a little bit more clear about where the dot product comes in. We established above that $\mathbf{A} = u_\alpha \mathbf{R}\hat{\mathbf{e}}_x+u_\beta\mathbf{R}\hat{\mathbf{e}}_y$ and that $\mathbf{A} = u_x\hat{\mathbf{e}}_x+u_y\hat{\mathbf{e}}_y$. Assuming (as it seems in your picture) that the basis vectors are orthonormal (that is, unit vectors and $\hat{\mathbf{e}}_x \cdot \hat{\mathbf{e}}_y = \hat{\mathbf{e}}_\alpha \cdot \hat{\mathbf{e}}_ \beta = 0$), we can pick out the x (y) component of $\mathbf{A}$ by taking its dot product with $\hat{\mathbf{e}}_x$ ($\hat{\mathbf{e}}_y$), since $$\mathbf{A}\cdot\hat{\mathbf{e}}_x = u_x \hat{\mathbf{e}}_x\cdot \hat{\mathbf{e}}_x + u_y\hat{\mathbf{e}}_y \cdot \hat{\mathbf{e}}_x = u_x$$ Now, using the other expression for $\mathbf{A}$, we get $$\begin{align*} u_x &= \hat{\mathbf{e}}_x \cdot \mathbf{A} \\ &= \hat{\mathbf{e}}_x \cdot (u_\alpha \mathbf{R}\hat{\mathbf{e}}_x+u_\beta\mathbf{R}\hat{\mathbf{e}}_y) \\ u_x &= u_\alpha (\hat{\mathbf{e}}_x \cdot \mathbf{R} \hat{\mathbf{e}}_x) + u_\beta (\hat{\mathbf{e}}_x \cdot \mathbf{R} \hat{\mathbf{e}}_y) \\ u_y &= u_\alpha (\hat{\mathbf{e}}_y \cdot \mathbf{R} \hat{\mathbf{e}}_x) + u_\beta (\hat{\mathbf{e}}_y \cdot \mathbf{R} \hat{\mathbf{e}}_y)\end{align*}$$ which can then be put into matrix form as I did above. If you're after a geometric interpretation, we can remove the reference to $\mathbf{R}$ altogether and just use the rotated representation of $\mathbf{A}$: $$\begin{align*} u_x &= u_\alpha (\hat{\mathbf{e}}_x \cdot \mathbf{R} \hat{\mathbf{e}}_x) + u_\beta (\hat{\mathbf{e}}_x \cdot \mathbf{R} \hat{\mathbf{e}}_y) \\ &= u_\alpha (\hat{\mathbf{e}}_x \cdot \hat{\mathbf{e}}_\alpha) + u_\beta (\hat{\mathbf{e}}_x \cdot \hat{\mathbf{e}}_\beta) \\ &= \hat{\mathbf{e}}_x \cdot (u_\alpha \hat{\mathbf{e}}_\alpha + u_\beta \hat{\mathbf{e}}_\beta) \\ &=\hat{\mathbf{e}}_x \cdot \mathbf{A} \\ u_y &= u_\alpha (\hat{\mathbf{e}}_y \cdot \mathbf{R} \hat{\mathbf{e}}_x) + u_\beta (\hat{\mathbf{e}}_y \cdot \mathbf{R} \hat{\mathbf{e}}_y) \\ &= u_\alpha (\hat{\mathbf{e}}_y \cdot \hat{\mathbf{e}}_\alpha) + u_\beta (\hat{\mathbf{e}}_y \cdot \hat{\mathbf{e}}_\beta) \\ &= \hat{\mathbf{e}}_y \cdot (u_\alpha \hat{\mathbf{e}}_\alpha + u_\beta \hat{\mathbf{e}}_\beta) \\ &=\hat{\mathbf{e}}_y \cdot \mathbf{A}\end{align*}$$
I'm being a bit long-winded, because there's all sorts of different expressions you could use depending on what you are trying to do, computationally or conceptually. Take your pick for the best expression - they are all equivalent - but take care that if your change-of-basis matrix is not a rotation matrix, or if your bases aren't orthonormal, the process is a bit more complicated.