Let's say I have this scalar product with the basis in both Cartesian and cylindrical coordinates. $$ê_y\cdotê_\rho$$
Do I need to convert one of them to the other basis, e.g. convert $ê_\rho$ to $cos\phiê_x+sin\phiê_y$ and then perform the scalar product $(0,1,0)\cdot(cos\phi, sin\phi,0)=(0,sin\phi,0)$?
Or how does it work?
The scalar product must be a scalar, so it is just $\sin \phi$. You can see this by converting one vector from cylindrical to cartesian co-ordinates or by using the co-ordinate free definition of the scalar product which is
$\overrightarrow{a} \cdot \overrightarrow{b} = |\overrightarrow{a}||\overrightarrow{b}| \cos \theta$
where $\theta$ is the angle between vectors $\overrightarrow{a}$ and $\overrightarrow{b}$. In you example $|\overrightarrow{a}|=|\overrightarrow{b}=1$ and $\theta = \frac{\pi}{2}-\phi$.