Hi I just want to confirm that this solution given for Chapter 12 Question 10(b) in the Answers Book is incorrect. The question asks for an expression for the Schwarzian derivative of inverse $f$. The solution says:
But instead of $1$ we should have:
$0 = \mathcal{D}(\operatorname{id}) = \mathcal{D}(f\circ f^{-1}) = \bigl((\mathcal{D}f)\circ f^{-1}\bigr)\cdot \bigl((f^{-1})'\bigr)^2 + \mathcal{D}(f^{-1}).$ As suggested in this duplicate question: Schwarzian derivative of inverse function..
For reference this is problem 10-17(a):
I just ran into this exercise myself and noticed the same thing. It's nice when MSE confirms a suspected error. For the record, I'm using 3rd Edition.
The Answer Book solution actually seems to contain two other substantial errors. Not Spivak's best outing!
Though the question is old, I'm offering an answer, less for the original poster and more for the next person that comes across this exercise:
Error 1: We need to first show that $f^{-1}$ is differentiable. The Answer Book fails to do this. Without this, neither 10(a) or 10(b) can be completed.
We are told that $\mathcal{D}f(x)$ exists for all $x$. From the definition of the Schwarzian derivative, this means that $f^\prime(x)$ exists and is $\neq 0$.
Thus $f$ is continuous.
Furthermore, suppose $f(a) = f(b)$ for some $a < b$. By Rolle's there exists some $x$ in the interval $(a,b)$ such that $f^\prime(x)= 0$.
Since we know $f^\prime(x)\neq 0$, this cannot be. Therefore, $f(a)\neq f(b)$ for any $a \neq b$, i.e. $f$ is one-one (and either increasing or decreasing).
To summarize, $f$ is a continuous one-one function with derivative $f^\prime(x) \neq 0$ for all $x$. Therefore, by the theorems in the chapter, $f^{-1}$ is continuous, one-one (increasing or decreasing) and differentiable for all $x$ in its domain.
Furthermore, $f^{-1\prime} \neq 0$, because if it were, $f$ would not be differentiable everywhere.
Now that we've demonstrated $f^{-1}$ is differentiable with $(f^{-1})^\prime \neq 0$, we can go about showing it's three times differentiable, as the book solution does. This will show that $\mathcal{D}f^{-1}(x)$ exists for all $x$.
Error 2: The Book answer for 10(a) lists the third order derivative of $f^{-1}$ as
$$(f^{-1})^{\prime\prime\prime}(x) = \frac{-[f^\prime(f^{-1}(x))]^3 f^{\prime\prime\prime}(f^{-1}(x)) + 3f^{\prime\prime}(f^{-1}(x))[f^\prime(f^{-1}(x))]^2}{[f^\prime(f^{-1}(x))]^7}$$
or, to simplify the expression a bit:
$$= \frac{-[f^\prime]^3 f^{\prime\prime\prime} + 3f^{\prime\prime}[f^\prime]^2}{[f^\prime]^7} \circ f^{-1}(x)$$
This is wrong. There seems to have been a calculation error, with the book missing a power of $f^{\prime\prime}$ in the numerator.
The correct expression, after simplifying is (I think):
$$(f^{-1})^{\prime\prime\prime}(x) = \frac{-f^{\prime\prime\prime}f^\prime + 3(f^{\prime\prime})^2} {[f^\prime]^5} \circ f^{-1}(x)$$
You can use this expression along with those for the first and second order derivatives to calculate part (b).
Error 3: As @helios321 points out, the answer for part (b) incorrectly states that $\mathcal{D}(f \circ f^{-1}) = 1$.
Actually,
$$\mathcal{D}(f \circ f^{-1}) = \mathcal{D}x = 0$$
To complete part (b) we can use this along with the identity from Problem 10-17(a) as suggested by the Answer Book, or simply use the derivatives of $f^{-1}$ and the definition of $\mathcal{D}$ to calculate $\mathcal{D}f^{-1}$.
Either way, (I believe) we should end up with $$\mathcal{D}f^{-1}(x)= \frac{3f^{\prime\prime2} - 2f^{\prime\prime\prime}f^\prime}{2f^{\prime 4}} \circ f^{-1}(x)$$