From Humphreys: A reflection is a linear operator $s$ on $V$ which sends some nonzero vector $\alpha$ to its negative while fixing pointwise the hyperplane $H_{\alpha}$ orthogonal to $\alpha.$ We may write $s=s_{\alpha}.$ The formula for the reflection is
$s_{\alpha}\lambda=\lambda-\frac{(\lambda,\alpha)}{(\alpha,\alpha)}\alpha.$
How to explicitly show that $s_{\alpha}$ is an orthogonal transformation, i.e. $(s_{\alpha}\lambda, s_{\alpha}\mu)=(\lambda, \mu)$ for all $\lambda, \mu \in V$?
Thanks!
Edit: Sorry for the wrong formula given above, it should be
$s_{\alpha}\lambda=\lambda-2\frac{(\lambda,\alpha)}{(\alpha,\alpha)}\alpha.$
Here is my attempt.
$(s_{\alpha}\lambda, s_{\alpha}\mu)=(\lambda - 2\frac{(\lambda,\alpha)}{(\alpha,\alpha)}\alpha , \mu - 2\frac{(\mu,\alpha)}{(\alpha,\alpha)}\alpha )=(\lambda, \mu)+(\lambda, - 2\frac{(\mu,\alpha)}{(\alpha,\alpha)}\alpha)+ (- 2\frac{(\lambda,\alpha)}{(\alpha,\alpha)}\alpha, \mu)+ (- 2\frac{(\lambda,\alpha)}{(\alpha,\alpha)}\alpha, - 2\frac{(\mu,\alpha)}{(\alpha,\alpha)}\alpha)$
and I don't know if it's correct.
Maybe, your formula is wrong. I know $s_{\alpha}(\lambda)=\lambda-2\frac{(\lambda,\alpha)}{(\alpha, \alpha)}\alpha$
First, show that any reflection is linear.
Finally, show that any reflection preserves norm.
Then we get : (Norm preserving + linear $\Rightarrow$ Inner product preserving.)
$\textbf{Solution:}$ Clearly, $s_{\alpha}(c\lambda)=c\lambda-2\frac{(c\lambda,\alpha)}{(\alpha, \alpha)}\alpha=cs_{\alpha}(\lambda)$.
Likewise, you can show that $s_{\alpha}(\lambda+\mu)=s_{\alpha}(\lambda)+s_{\alpha}(\mu)$ easily. (Use linearity of inner product.)
$||s_{\alpha}(\lambda)||^2=||\lambda||^2 +4\frac{(\lambda,\alpha)^2}{(\alpha,\alpha)^2}||\alpha||^2-4\frac{(\lambda,\alpha)}{(\alpha,\alpha)}(\lambda,\alpha)=||\lambda||^2$. Hence, reflection is norm-preserving and linear.
$\textbf{Claim}:$ If $T$ is a linear operator on $V$ and norm-preserving, $T$ preserves inner product.
Suppose that $T$ is linear and $||T(v)||=||v||$ for every $v\in V$. Then
$||u-v||^2=||T(u-v)||^2=(T(u-v),T(u-v))=(Tu-Tv,Tu-Tv)$
= $(Tu,Tu)+(Tv,Tv)-2(Tu,Tv)=||Tu||^2+||Tv||^2-2(Tu,Tv)=||u||^2+||v||^2-2(Tu,Tv)$
Therefore we have $$||u-v||^2=||u||^2+||v||^2-2(Tu,Tv).$$
Thus,
$$(u,v)=(Tu,Tv)$$ for every $u,v \in V.$