Currently, I am trying to understand the solution of a problem in my pattern recognition class, and I am stuck at a particular step, which looks at the second order Taylor expansion approximation. Here is the step in question: Solution manual's step
I have even asked my professor for help on how the solution manual got from one to the other. He's the one that stated that it is a Taylor series expansion evaluated at 0. In his solution (for the first part), he made the change of variables that
$\tfrac{h_n^2}{\sigma^2} = x^2$, and that $x^2$ = y,
making the derivative easy to compute. for the taylor expansion, he used f(0) + f '(0), which gives exactly the approximation shown in the solutions manual. My problem is, why are we allowed to simply make a change of variables to what we want and how we want? If we do not do a change of variable, and calculate the derivative from the equation as is, we would get
1 - $\tfrac{h_n}{(\sigma^2(x^2/\sigma^2 + 1)^{3/2})}$,
and when evaluated at small hn, we'd get 1 - 0. These are two completely different answers, which is what baffles me. And I'm having the same issue with the second Taylor expansion in the solution, as it only works if I change the whole power of e to y, and take the derivative with respect to y, and evaluate at 0.
Could anyone please help me understand why we're allowed to change the variables in whichever fashion seems to work without having any consequences? Why are the two approximations valid? The whole purpose of the problem in question is to show that a particular equation is approximately equal to another, and the rest of the problem relies on these Taylor expansions. I hope that this is a valid question to ask, I have been on this on my own for at least a good 5 hours.
Edit: if anyone is actually interested in working out the full problem (just know that it took me a week to do a), b), and half of c): Full problem statement
The changes of variable do not change the results. The following Taylor series are expanded at zero in the indicated variable to the indicated order: \begin{align*} (h,3)& & \frac{1}{\sqrt{1+ h^2/\sigma^2}} = 1&-\frac{1}{2\sigma^2}h^2 + \cdots \text{,} \\ (x,3)& & \left.\frac{1}{\sqrt{1+ x^2}} \right|_{x \rightarrow h/\sigma} = 1 &- \left. \frac{1}{2} x^2 + \cdots \right|_{x \rightarrow h/\sigma} \\ && {} = 1 &- \frac{1}{2} \cdot \frac{h^2}{\sigma^2} + \cdots \text{, and} \\ (y,1)& & \left.\frac{1}{\sqrt{1+ y}} \right|_{y \rightarrow h^2/\sigma^2} = 1 &- \left. \frac{1}{2} y + \cdots \right|_{y \rightarrow h^2/\sigma^2} \\ && {} = 1 &- \frac{1}{2} \cdot \frac{h^2}{\sigma^2} + \cdots \text{.} \end{align*}
(Note the expansions are to order $3$ in the first two, since there are no odd order terms in these series.)
Making those changes of variables does not change the results. It does make the Taylor series a little easier to compute by reducing the complexity of the derivatives (and pushing some computations out of this step via implicit differentiation). \begin{align*} \frac{\mathrm{d}}{\mathrm{d}h} \frac{1}{\sqrt{1+ h^2/\sigma^2}} &= -\frac{1}{2} \left( 1+ h^2/\sigma^2 \right)^{-3/2} \cdot 2h/\sigma^2 \text{.} \\ \frac{\mathrm{d}}{\mathrm{d}y} \frac{1}{\sqrt{1+y}} &= -\frac{1}{2} (1+y)^{-3/2} \text{.} \end{align*} Of course, for a Taylor expansion centered at $h = 0$ (and since $y = h^2/\sigma^2$, also at $y = 0$), we always recall that we are computing $\frac{f^{(n)}(0)}{n!}$, so we need to evaluate these at $h = 0$ and $y=0$. So the first one is $0$ and the second is $-1/2$. We can see this in the table above. There is no linear term in the expansions for $h$ or $x$ -- the first nonconstant term is quadratic. There is a linear term for $y$, but remember that this corresponds to the quadratic term in $h$ and $x$.
The first nonzero derivative for the expansion in $h$ is the second derivative. $$ \frac{\mathrm{d}^2}{\mathrm{d}h^2} \frac{1}{\sqrt{1+ h^2/\sigma^2}} = \frac{2 h^2-\sigma ^2}{\left(h^2+\sigma ^2\right)^2 \sqrt{\frac{h^2+\sigma^2}{\sigma ^2}}} \text{.} $$ Evaluating at $h = 0$, we find $-1/\sigma^2$. Combined with the $\frac{1}{2!}$ from the recipe for the Taylor series, we get the term shown in the table above, $\displaystyle \frac{-1}{2\sigma^2} h^2$.