Let $U \subset \mathbb{R}^m$ be open and $f \colon U \rightarrow f(U) \subseteq \mathbb{R}^n$ be a homeomorphism.
I think this means that $f(U)$ is not necessarily an open set but $f$ is an open map with respecet to the subspace topology of $f(U)$ in $\mathbb{R}^n$. Is this correct?
Now my question is, if $m=n$, why can we conclude that $f(U)$ is an open subset of $\mathbb{R}^m$? And why is this wrong in general for $m \neq n$?
You are correct: to say $f:U\to f(U)$ is a homeomorphism means that it is an open map with respect to the subspace topology on $f(U)$, so that does not automatically mean $f(U)$ is open in $\mathbb{R}^n$
The fact that $f(U)$ is open if $m=n$ is a hard theorem, known as "invariance of domain". You can find a proof in many algebraic topology texts; for instance, it is Theorem 2B.3 of Hatcher's Algebraic Topology.
If $m\neq n$, there are very easy counterexamples. For instance, the map $f:\mathbb{R}\to\mathbb{R}^2$ defined by $f(x)=(x,0)$ is a homeomorphism to its image but its image is not open in $\mathbb{R}^2$.