For spherical coordinates, for instance, you can derive the unit vector $\hat\phi$, for instance, by taking the derivative of the position vector with respect to $\phi$ and normalizing it.
Why does taking the derivative of the position vector with respect to one of the coordinates give you a vector in the direction of increase of that coordinate?
Why is it important that a position vector is used, as opposed to some other kind of vector?
I am not sure that this answer is what you want. If not, please advise and I will gladly delete it. For simplicity, consider polar coordinates in $\mathbb R^2.$ The unit vectors are actually $\textit{tangent}$ vectors, in the following sense:
Fix $p=(x,y)\in \mathbb R^2$. In rectangular coordinates, $e_x$ and $e_y$ are unit vectors in the $x$ and $y$ directions, respectively. But they can also be considered as measuring the velocity of a particle that moves tangent to, i.e., in their direction. Then, $(e_1,e_2)$ span the "tangent" vector space to $\mathbb R^2$ at $p=(x,y).$ The reason for taking this perspective will be clear from what follows.
If we locate $p$ on a circle of radius $r$ and an angle $\theta$, measured from the $x$-axis to the segment from the origin to $p$, and try to measure the velocity of a particle that moves on circles (as $r$ changes) and through angles (as $\theta$ changes), we run into the problem that the displacement of the particle is not being measured "on a straight line". As it moves, measured in relation to the circle of radius $r$, it instantaneously changes its direction. So we are forced to consider tangents of position vectors. So, we want to describe the motion of the particle with respect to these new position coordinates $e_r$ and $e_{\theta}$. And the point is that they are changing direction as the particle changes position.
The same thing can be done with the vectors in rectangular coordinates: $v=(x,y)$ and so $\partial v_x=(1,0)=e_x$ and $\partial v_y=(0,1)=e_y$ so in rectangular coordinates, the derivative of position in the $x$-direction the $constant$ vector $e_x$ and similarly for the $y$-direction. In polar coordinates, the situation is quite different:
Since we have fixed $p\in \mathbb R^2$, it makes sense to look for a change of basis matrix between the two coordinate systems. The relation between the two is given by $x=r\cos\theta$ and $y=r\sin \theta,$ which is not linear but since we only want to see the relation between the systems $locally$, we use the Jacobian matrix of this function at $p$:
$\mathcal J=\begin{pmatrix} \cos\theta &-r\cos\theta \\ \sin\theta& r\cos\theta \end{pmatrix}$
so since
$\begin{pmatrix} \cos\theta &-r\cos\theta \\ \sin\theta& r\cos\theta \end{pmatrix} \begin{pmatrix} \cos\theta\\ -\frac{1}{r}\sin\theta\end{pmatrix}=\begin{pmatrix} 1\\ 0\end{pmatrix},$ we have
$e_x=(\cos\theta)e_r-\frac{1}{r}(\sin\theta)e_{\theta}$ and similarly $e_y = (\sin\theta) e_{r} + \tfrac{1}{r}(\cos\theta) e_{\theta}$
which are exactly the equations we get when we formally take derivatives applying the chain rule.