This is Lemma 6 in page 14 of Milnor's Topology from the Differentiable Viewpoint.
Lemma 6. Any smooth map $g:D^n \to D^n$ has a fixed point.
Proof. Suppose $g$ has no fixed point. For $x \in D^n$, put $$ u(x)=\frac{x-g(x)}{|x-g(x)|}, t(x)=-x \cdot u(x)+\sqrt{1-x \cdot x+(x\cdot u(x))^2 } $$ and define $f(x)=x+t(x)u(x)$. Then $f$ is a smooth retraction of $D^n$ onto $S^{n-1}$, contradicting the above lemma.
There is a figure (below) in the book. But I can't see how the formula of $f$ come from. Also, I can't see that $t$ and $f$ are well-defined. How do I have derive the formula?

We're trying to create a retraction that maps points $g(x)$ to points $f(x)$. The idea is that since there are no points such that $g(x) = x$, we can always find a line from $g(x)$ to $x$, and extend this line to meet the boundary at $f(x)$.
Intuitively, such a process ought to be continuous: wiggling the point $x_0 \in D^n$ should wiggle $g(x_0)$ by a small amount (since $g$ is smooth), and should hence wiggle $f(x_0)$ by a small amount (since $f$ is a line that's joining two smoothly wiggling points, the wiggle in the endpoint of $f$ as it touches the disk should be small).
So here, $u(x)$ is the normalized direction vector from $g(x)$ to $x$: we take the direction $g(x) - x$ and then normalize with $|g(x) - x|$ to create $u(x)$.
Next, we need to extend the line starting at $x$, pointing in direction $u(x)$, with magnitude that will take it to the surface of the disk $D^n$.
Notice that such a vector will be of the form $\texttt{start} + \texttt{dir} \cdot \texttt{length}$, where:
So that gives us the formula for $f(x) \equiv x + u(x)t(x)$
To derive the formula for $t(x)$, notice that $f(x)$ is a point on the sphere, so we need $||f(x)||^2 = 1$. From this, we get:
\begin{align*} ||f(x)||^2 &= 1 \\ ||x + t u||^2 &= 1 \\ ||x||^2 + ||tu||^2 + 2(x \cdot t u) &= 1 \\ ||x||^2 + t^2||u^2|| + 2 (x \cdot u) t - 1 &= 0 \\ t^2||u^2|| + 2 (x \cdot u) t - 1 + ||x||^2 &= 0 \\ t^2 \cdot 1 + 2 (x \cdot u) t - 1 + ||x||^2 &= 0 \qquad \text{$||u|| = 1$ since $u$ is unit vector}\\ \end{align*}
Solving the quadratic for $t^2$ with $a = 1, b = 2 (x \cdot u), c = ||x^2 - 1||$, we arrive at the desired expression for $t$.