In the book of Linear Algebra by Greub, at page 237, it is asked to prove that
The characteristic polynomial of a proper rotation satisfies the relation $f(\lambda) = (-\lambda)^n f(\lambda^{-1})$
where a rotation defined as an isometry on a finite dimensional real inner product space $E$.
I have tried the following:
$$(-\lambda)^n det(\phi - \lambda^{-1}i) = det(-\lambda \phi + i),$$
and we know that $det\phi = \pm 1$, so in a sense the above equation is similar to $\det(\phi - \lambda i)$, however the entries of the matrix of $\phi$ and $i$ are obviously different, and applying the definition of determinant given in the book, I couldn't arrive anywhere, so I would appreciate any help or hint.
The problem as originally stated by our OP onurcanbektos can't be quite right, as is illustrated by the following example which is based upon the identiy map $I$ on $\Bbb R^3$: consider $\phi = -I$ on $\Bbb R^3$: the characteritic polynomial of $\phi$ is
$f(\lambda) = \det (-I - \lambda I) = \det((1 + \lambda)(-I)) = (1 + \lambda)^3 \det(-I) = -(1 + \lambda)^3; \tag 1$
then
$f(\lambda^{-1}) = -(1 + \lambda^{-1})^3, \tag 2$
and
$(-\lambda)^3 f(\lambda^{-1}) = -\lambda^3 (-(1 + \lambda^{-1})^3) = (1 + \lambda)^3; \tag 3$
here we have an example that
$f(\lambda) = -(-\lambda)^3 f(\lambda^{-1}); \tag 4$
now if our isometry is $I$, we find
$f(\lambda) = \det (I - \lambda I) = \det((1 - \lambda)I) = (1 - \lambda)^3 \det (I) = (1 - \lambda)^3, \tag 5$
and
$f(\lambda^{-1}) = (1 - \lambda^{-1})^3, \tag 6$
so that
$(-\lambda)^3f(\lambda^{-1}) = (-\lambda)^3 (1 - \lambda^{-1})^3 = -(\lambda(1 - \lambda^{-1}))^3$ $= -(\lambda - 1)^3 = (1 - \lambda)^3 = f(\lambda). \tag 7$
If we compare (4) and (7) we might speculate that there is a factor or other information related to $\det \phi$ which is missing from the equation as stated in the text of the question,
$f(\lambda) = (-\lambda)^n f(\lambda^{-1}); \tag 8$
in fact, I researched Greub's book, Linear Algebra, and discovered that the actual problem quoted here, problem 9 on p. 237, reads
$"$9.) Prove that the characteristic polynomial of a proper rotation satisfies
$f(\lambda) = (-\lambda)^n f(\lambda^{-1})." \text{[italics mine.]} \tag 9$
Greub defines $\phi$ to be a proper rotation if it is an isometry with $\det \phi = 1$.
Based on this understanding, problem (9), p. 237 of Greub may be solved as follows:
$\phi$ is an isometry of the real inner product space $E$ if and only if it preserves inner products; that is, for $x, y \in E$ we have
$\langle \phi x, \phi y \rangle = \langle x, y \rangle; \tag {10}$
this of course implies
$\langle x, \phi^T \phi y \rangle = \langle x, y \rangle, \tag{11}$
and also
$\langle \phi^T \phi x, y \rangle = \langle x, y \rangle; \tag{12}$
as is well-known, (11) and (12) imply
$\phi^T \phi = I = \phi \phi^T, \tag{13}$
which shows that $\phi$ is invertible and that
$\phi^{-1} = \phi^T; \tag{14}$
such transformations are also called orthogonal. Since
$\det \phi^T = \det \phi, \tag{15}$
a well-known property of determinants, it follow from (13) that
$(\det \phi)^2 = (\det \phi^T)(\det \phi) = \det \phi^T \phi = \det I = 1; \tag{16}$
that is,
$\det \phi = \pm 1; \tag{17}$
we also have
$\det \phi^{-1} \det \phi = \det \phi^{-1} \phi = \det I = 1; \tag{18}$
combining this with (15) and (17) we find
$\det \phi^T = \det \phi^{-1} = \det \phi = \pm 1; \tag{19}$
if $\phi$ is proper, we choose the $+$ sign in (19).
With these preliminaries completed, we thus have:
$(\phi - \lambda I)^T = \phi^T - \lambda I, \tag{20}$
whence
$f(\lambda) = \det(\phi - \lambda I) = \det ((\phi - \lambda I)^T) = \det (\phi^T - \lambda I); \tag{21}$
then from (14),
$f(\lambda) = \det (\phi^T - \lambda I) = \det (\phi^{-1} - \lambda I)$ $= \det (\phi^{-1}(I - \lambda \phi)) = \det \phi^{-1} \det(I - \lambda \phi) = \det(I - \lambda \phi), \tag{22}$
where we have exploited the $+1$ case of (19); we thus conclude:
$f(\lambda) = \det(I - \lambda \phi) = \det (-\lambda I)\det(\phi - \lambda^{-1}I) = (-\lambda)^nf(\lambda^{-1}), \tag{23}$
establishing (8) for proper rotations.
In the event that $\phi$ is not a proper rotation, i.e., that $\det \phi = -1$, we can go back to (22) and pick up the derivation to find
$f(\lambda) = \det (\phi^T - \lambda I) = \det (\phi^{-1} - \lambda I)$ $= \det (\phi^{-1}(I - \lambda \phi)) = \det \phi^{-1} \det(I - \lambda \phi) = -\det(I - \lambda \phi), \tag{24}$
from which it follows as in (23),
$f(\lambda) = -(-\lambda)^n f(\lambda^{-1}); \tag{25}$
we note these formulas are consisent with (4) and (8).