I define a vector as any object $(a_i,a_j, a_k)$ such that it transforms the same way as the coordinates themselves. That is if $x'_i = R_{ij}x_j$, then $a'_i = R_{ij}a_j$ (using the Einstein convention of summing over repeated indices). Please correct me if this is not a proper definition.
I now want to take the cross product of two vectors, $\vec{c} = \vec{a}\times \vec{b}$ and prove (by this definition) that the output is also a vector. Using the Levi-Civita symbol, we have
$c'_i = R_{ij}c_j = det(R) R_{ij}\epsilon_{jlm}a_l b_m$
But this should somehow be the same as the cross product of the transformed $\vec{a}$ and $\vec{b}$. That is $c'_i = \epsilon_{ijk}a'_jb'_k = \epsilon_{ijk}R_{jl} a_l R_{km}b_m$
How do I reconcile these two? I know I should get $\rm{det(R)}$ in the second expression somehow but I can't see how to express it in this notation.
Suppose $\mathbf c = \mathbf a \times \mathbf b$. Then, since $\mathbf a$ and $\mathbf b$ are vectors and $\varepsilon_{ijk}$ is a rank $3$ tensor, \begin{align*} a_i' &= R_{ij} a_j \\ b_i' &= R_{ij} b_j \\ \varepsilon_{ijk}' &= R_{ip}R_{jq}R_{kr} \varepsilon_{pqr}. \end{align*} Now under a coordinate transformation, \begin{align*} c_i& = \varepsilon_{ijk} a_j b_k \\ \implies c_i' &= \varepsilon_{ijk}' a_j' b_k' \\ &= R_{ip} R_{jq} R_{kr} \varepsilon_{pqr }R_{jm}a_m R_{kn}b_n \\ &= R_{ip} \delta_{qm} \delta_{rn} \varepsilon_{pqr} a_m b_n \tag{$*$} \\ &= R_{ip} \varepsilon_{pqr}a_q b_r \\ &= R_{ip} c_p, \end{align*} where $(*)$ follows since $R_{jq} R_{jm} = [\mathbf R^T]_{qj} \left[\mathbf R\right]_{jm} = \left[\mathbf R^T \mathbf T\right]_{qm} = \delta_{qm}$.