Let $M_2(\mathbb R)$ be the set of all $2\times 2$ real matrices and $U$ be the set of all invertible upper triangular matrices ,i.e.,$ U=\{\begin{bmatrix} {x_1}&{x_2}\\{x_3}&{x_4}\end{bmatrix}\in M_2(\mathbb R): x_3=0 \ and\ x_1x_4\neq0\}. $
Now the question is 'Is $U$ open or not w.r.t. usual topology?'
Here if we suppose $U\subseteq\mathbb R^3$ then $U=\{(x_1,x_2,x_4)\in \mathbb R^3:x_1x_4\neq0\}$ and this is an open set as $U=f^{-1}((-\infty,0)\cup(0,\infty))$ where $f:\mathbb R^3\to \mathbb R $ such that $f(x_1,x_2,x_4)=x_1x_4$ and $f$ is a continuous function. So in this case $U$is an open set.
Now if we suppose $U\subseteq\mathbb R^4$ then $U=\{(x_1,x_2,x_3,x_4):x_3=0\ and\ x_1x_4\neq0\}\subset \mathbb R^3\times \{0\}.$ So in this case $U$ can't be open,right?
So what will be the right answer?
Any help is appreciated. Thank you.
As you essentially showed, $U$ is an open subset of the space of upper triangular matrices, but not an open subset of the space of all matrices.
A set is never open per se. You always have to mention the ambient topological space you want to consider ("is open in" or "is an open subset of").