How can I prove
$\alpha(W)=W\Rightarrow\alpha(W^\perp)=W^\perp$ where W is a subspace of $\mathbb{R}^n$?
I tried as below..
We have to show
If $x\in W^\perp$, then there exists $y\in W^\perp$ such that $\alpha(y)=x$
Since $x,y\in W^\perp$, $x\cdot w=0$ and $y\cdot w=0$ for all $w\in W$
I don't know what to do next.
And What is the domain of $\alpha$?
First, the domain of $\alpha$ is $\mathbb{R}^n$.
In fact, it is not true generally. You can consider $\mathbb{R}^2$ and $W$ the subspace spanned by $(1,0)$. Let $\alpha$ defined by $\alpha(x,y)=(x+y,y)$. It is clear that $\alpha(W)=W$ and $\alpha(W^{\perp})\cap W^{\perp}=\{0\}$.
It can be proved under the condition when $\alpha$ preserves the inner product, i.e., $\alpha(u)\cdot \alpha(v)=u\cdot v$ for all $u,v\in\mathbb{R}^n$. Let $w_1,\cdots,w_k$ be a basis of $W$ and $u_1,\cdots,u_{n-k}$ be a basis of $W^{\perp}$. Then $w_1,\cdots,w_k,u_1,\cdots,u_{n-k}$ form a basis of $\mathbb{R}^n$. Let $A$ be the matrix of $\alpha$ under this matrix. Then $A$ is an orthogonal matrix and it looks like \begin{pmatrix} A_1 & B \\ 0 & C \end{pmatrix} We need to show that $B=0$. For $u_j$, note that $\alpha(u_j)=B_{1j}w_1+\cdots+B_{kj}w_k+C_{1j}u_1+\cdots+C_{n-k,j}u_{n-k,j}$. Since $\alpha(u_j)\cdot \alpha(w_i)=0$ for all $i$, we have $B_{lj}=\cdots=B_{kj}=0$. This complete the proof.