We define the attitude error function $\Psi(R, R_d)$ over $SO(3)$ as : $$ \Psi(R, R_d) = \frac{1}{2}tr(I - R_d^{\top}R)$$ This acts as a metric to define distance between two rotation matrices which otherwise can't be calculated using euclidean vector space subtraction.
But we also know that $\Psi = \mathbb{cos(\phi)}$, where $\mathbb{\phi}$ is the rotation angle about the rotation axis as obtained through the axis-angle representation of rotation matrices.
I wanted to ask if $\Psi$ is actually a metric (in a formal sense) or not? Meaning does it satisfy the triangle inequality : $$ \Psi(I, R) \le \Psi(I,R_d) + \Psi(R_d,R)$$
I tried working it out, also did brute force expansion, but couldn't really come to a conclusion. Since $\Psi$ corresponds to the rotation angle, I feel like this should be true as we can see by keeping an axis fixed, say $e_1$ and perform 2 successive rotations about the same axis corresponding to $R_d$ and $R$ respectively.
I have very elementary knowledge regarding the above. Any help is appreciated.
As I noted in my comment, your $\Psi$ is not a metric. Specific examples violating the triangle inequality are given by the triples $I, R_\phi, R_{2\phi}$, for any given $\phi\in (0, \frac{\pi}{2})$, where $I$ is the identity and $R_\psi$ is the rotation by the angle $\psi$ around the $z$-axis.
However, $\sqrt{\Psi}(A,B)$ is a metric since it equals $||A-B||_F$ where the norm is the restriction of the Frobenius norm on the space of all 3-by-3 matrices (the same works in all dimensions). Recall that the Frobenius norm $||\cdot||_F$ on the vector space of $n\times n$ matrices is the norm associated with the bilinear form $$ \langle A, B\rangle= tr(A^TB), $$ $$ ||A||_F= \sqrt{tr(A^TA)}. $$
From the fact that the metric defined by the Frobenius norm satisfies the triangle inequality, you get the inequality $$ \Psi(A,C)\le [\Psi^{1/2}(A,B)+\Psi^{1/2}(B,C)]^2, $$ for all orthogonal matrices $A, B, C$.
Nevertheless, your function $\Psi$ does satisfy an interesting "metric" property, it is what's called a conditionally negative semidefinite kernel:
Definition. Given an infinite set $X$ (such as the set of all orthogonal 3-by-3 matrices), a function $\Psi: X\times X\to {\mathbb R}$ is conditionally negative semidefinite if it determines a bilinear form of the signature $(1,\infty)$ on the space $Map_F(X, {\mathbb R})$ consisting of all maps $X\to {\mathbb R}$ with finite support. This space has the basis $\{\chi_x; x\in X\}$, where $\chi_x(x)=1, \chi_x(y)=0$ for all $y\ne x$. Then the bilinear form on $Map_F(X, {\mathbb R})$ is determined by its value on the basis of this space which is given by the formula $$ \langle \chi_x, \chi_y\rangle= \Psi(x,y). $$
The fact that your function $\Psi$ has this property has nothing to do with orthogonal matrices: Given any subset $X$ of a Euclidean space $E$ (of any dimension, even a Hilbert space), the function $$ \Psi(x,y)= \left ( dist_E(x,y) \right)^2 $$ is a conditionally negative semidefinite kernel on $X$. One can prove, furthermore, that if $X$ is a smooth submanifold of $E$ of dimension $>0$, then $\Psi$ is never a metric on $X$.