Let $U$ be an open subset of the Euclidean space $E^3$, $\mathcal{T}$ is a distance-preserving transformation from $U$ to itself. Prove that $\mathcal{T}$ can be extended to an isometry(i.e. a distance-preserving transformation) on $E^3$.
I aimed to find an isometry on $E^3$ that can be restricted to $U \subset E^3$ but failed, and I have no idea where to start. Thanks for any help!
In the form you state it, this is not true. You have to assume that $U$ is connected, otherewise you could use restrictions of different isomoetries of $E^3$ to different connected components. In fact, the best way to phrase the statement is that any distance preserving map $f$ between connected open subsets of $E^3$ is the restriction of a (uniuqe) Euclidean motion. In this form, you can prove it in a similar way as the characterization of Euclidean motions in terms of translations and orthogonal maps:
Applying translations, you can assume without loss of generality that $0$ lies in $U$ and in $V$ and that $f(0)=0$. Since $U$ is open and $0\in U$, it contains a ball of radius $\epsilon$, so there are $x_1,x_2,x_3\in U$ which are mutually orthogonal and all have length $\epsilon$. Then also $f(x_1),f(x_2),f(x_3)$ are mutally orthogonal and all have length $\epsilon$. Thus there is a unique orthgonal linear map $A$ on $\mathbb R^3$ such that $f(x_i)=Ax_i$ for $i=1,2,3$.Now as in the standard proof, you conclude from the fact that $f$ is an isometry on $U$ that for $y\in U$, you get $\langle f(y),f(x_i)\rangle=\langle y,x_i\rangle$. Now by construction $y=\sum_i\tfrac1{\epsilon^2}\langle y,x_i\rangle x_i$, and $f(y)= \sum_i\tfrac1{\epsilon^2}\langle f(y),f(x_i)\rangle f(x_i)$, which shows that $f(y)=Ay$. So $f$ is the restriction of a linear isometry of $E^3$ and bringing the translations back in, the general statement follows.