Let $X$ and $Y$ be CW complexes. Fix a ring $R$.
Künneth formula says
$$ H^k(X\times Y, R)\cong \bigoplus_{r+s=k} H^r(X,R)\otimes H^s(Y,R)$$
I am not able to see a reference where it is mentioned on which ring they are taking tensor product.
I think it is over the ring $R$. As coefficients are taken in $R$, now these $H^r(X,R)$ are $R$ modules (are they free or something like that?). Thus, one can talk about tensor product of $R$ modules $H^r(X,R)\otimes_R H^s(Y,R)$.
Is this correct? Am I misunderstanding anything?
First of all, that's not the statement of the Künneth theorem in general, but it is the statement if $R$ is a field. Instead, if $R$ is a principal ideal domain and one of $H^*(X; R)$ and $H^*(Y; R)$ is of finite type, then there is a short exact sequence
$$0 \to \bigoplus_{r + s = k}H^r(X; R)\otimes H^s(Y; R) \to H^k(X\times Y; R) \to \bigoplus_{p + q = k + 1}\operatorname{Tor}_R(H^p(X; R), H^q(Y; R)) \to 0$$
which splits non-canonically; see Theorem $5.5.11$ of Spanier's Algebraic Topology.
But yes, $H^r(X; R)$ and $H^s(Y; R)$ are both $R$-modules, so the tensor product is taken over $R$. However, it is not necessarily the case that they are free $R$-modules, e.g. $H^2(\mathbb{RP}^2; \mathbb{Z}) \cong \mathbb{Z}_2$. If they were free modules (as is necessarily the case when $R$ is a field), the $\operatorname{Tor}$ term would be zero and you'd get the statement as in your question.