I learned that the subspace of $\mathbb R^2$ can only be the origin or a $\mathbb R$ space that pass through the origin.
I also learned that the subspace of $X$ is any subset that contains an open set.
The two definitions seem to be contradictory.
My question is, what is rigorous definition for subspace?
The word "subspace" on its own doesn't mean much and depends on the context. It can mean:
and more. One could use the same term for normed, Banach, inner product, Hilbert, affine, measurable, probability subspaces. Even schemes or topoi. The list is just huge, close to endless.
All those have precise definitions, often unrelated to other structures. And so you need to understand the context, in order to understand what "subspace" means.
Here, the author most likely refers to linear subspace. Note that $\mathbb{R}^2$ itself is also a linear subspace of $\mathbb{R}^2$.
Unlikely anyone defined "subspace" like that. In the context of metric and/or topology, the term "subspace" literally means any subset. Subset that contain an open subset are typically called a subset with nonempty interior.