Let $PSL(2,\mathbb{R})$ be the Projective Special Linear Group and $PSO(2)$ be the Projective Special Orthogonal Group.
It is well-known that $PSL(2,\mathbb{R})/PSO(2)$ can be identified with the upper half-plane $\mathbb{H}$.
Let $g,h$ be two elements of $PSL(2,\mathbb{R})$, How can the hyperbolic distance between $gPSO(2)$ and $hPSO(2)$ be computed?
On
It depends on what you mean by an explicit formula:
Every $n\times n$ real matrix $A$ admits a QR decomposition as the product $$ A= QR $$ where $Q$ is orthogonal and $R$ is upper triangular. For us, it is more convenient to use the RQ decomposition. (It is easy to go between the two, especially for invertible matrices since $A^{-1}=R^{-1}Q^{-1}$.) The decomposition is algorithmic, see the linked Wikipedia Article.
Now, suppose that $A\in SL(2, {\mathbb R})$; then you get a decomposition $$ A=RQ. $$ This decomposition is unique as long as you assume that both diagonal entries of $R$ are positive.
The quotient $G/K=SL(2, {\mathbb R})/SO(2)$ of course is canonically identified with the quotient $PSL(2, {\mathbb R})/PSO(2)$, but I find it easier to work with. Then the projection of $A\in SL(2, {\mathbb R})$ to $G/K$ can be identified with the matrix
$$
R=\left[\begin{array}{cc}
a&b\\
0&a^{-1}\end{array}\right], a>0, b\in {\mathbb R}.
$$
This matrix acts on the upper half-plane $H^2$ (in the complex plane) by the formula that you surely know:
$$
z\mapsto a^2z+ab.
$$
The orthogonal matrix $Q$ in our decomposition stabilizes the point $i=\sqrt{-1}\in H^2$, hence, under the identification of $G/K$ with $H^2$, the matrix $R$ is identified with the point
$$
z_R=ab+ i a^2.
$$
Now, given two matrices $A_1=R_1Q_1, A_2=R_2Q_2$, the distance between their cosets in $G/K$ is the same as the hyperbolic distance between the points $z_{R_1}, z_{R_2}$ which you can find, for instance, in the Wikipedia article on the upper half-plane model:
$$
d_{H^2} (x_1+i y_1, x_2+ i y_2)
= \operatorname{arcosh} \left( 1 + \frac{ {(x_2 - x_1)}^2 + {(y_2 - y_1)}^2 }{ 2 y_1 y_2 } \right)$$
In terms of the matrices $R_1, R_2$, the formula becomes:
$$
d(z_{R_1}, z_{R_2})=
\operatorname{arcosh} \left( 1 + \frac{ {(a_2b_2 - a_1b_1)}^2 + {(a^2_2 - a^2_1)}^2 }{ 2 a^2_1 a^2_2 } \right)
$$
Like it or not, this is the formula you asked for.
If you do not like using the RQ-decomposition, you can still identify the coset of the given matrix $$ A=\left[\begin{array}{cc} a&b\\ c&d\end{array}\right], $$ with the point $$ z_A= \frac{ai+b}{ci+d}\in H^2. $$ Then $$ d(A_1 SO(2), A_2 SO(2))= d_{H^2}(z_{A_1}, z_{A_2}). $$ Once you write down this formula in all the detail in terms of the matrix coefficients of $A_1, A_2$ (I will not attempt to do so), you will realize how useless it is.
The hyperbolic distance between $xPSO(2)$ and $yPSO(2)$ is equal to the hyperbolic distance between the identity coset $PSO(2)$ and $x^{-1}yPSO(2)$. So it suffices to describe the distances from the basepoint stabilized by $PSO(2)$ to another arbitrary point $gPSO(2)$.
I'll lean on some standard linear algebra here. Using the singular value decomposition, we can write $g=kak'$ for $k,k' \in PSO(2)$ and $a$ a diagonal matrix. The nonzero entries of $a$ are the singular values of $g$. These have to be multiplicative inverses and let's denote them by $\sigma \ge 1$ and $\sigma^{-1}$.
The distance from the basepoint to $gPSO(2)$ equals the distance to $aPSO(2)$. That's a pretty easy distance to compute. In particular, if your metric on the upper half plane $\mathbb{H}^2$ has constant curvature $-1$, then we get $$ d(PSO(2),gPSO(2))=2 \log \sigma .$$ Therefore the distance between $xPSO(2)$ and $yPSO(2)$ is easily computed from the singular values of $x^{-1}y$.
This formula can be viewed as a special case of a larger story; in particular the Cartan decomposition of symmetric spaces.