Let $x \in l_2$ and $J(x) = \sum_{k = 1}^{+\infty} x_k x_{k + 1}$. Find $DJ(u)$ and $D(DJ(u))$.
Attempted solution
Since $x \in l_2$, then $\sum_{k = 1}^{+\infty}x_k < \infty$. Another fact: $\|x\|_{l_2} = (\sum_{k=1}^{+\infty}x_k^{2})^{\frac{1}{2}}$.
By the definition of Frechet derivative: $$ \lim_{h\to0} \frac{|J(u + h) - J(u) - (DJ(u))(h)|}{\|h\|_{l_2}} = \lim_{h\to0} \frac{|\sum_{k = 1}^{+\infty}(u_k + h_k)(u_{k + 1} + h_{k + 1}) - \sum_{k = 1}^{+\infty}u_ku_{k+1} - (DJ(u))(h)|}{\|h\|_{l_2}} = \lim_{h\to0} \frac{|\sum_{k = 1}^{+\infty}u_kh_{k+1} + \sum_{k = 1}^{+\infty}u_{k + 1}h_k + \sum_{k = 1}^{+\infty}h_kh_{k + 1} - (DJ(u))(h)|}{\|h\|_{l_2}} $$
Now I am quite tempted to say that the following is the Frechet derivative
$$(DJ(u))(h) = \sum_{k = 1}^{+\infty}u_kh_{k+1} + \sum_{k = 1}^{+\infty}u_{k + 1}h_k$$
and then provide an extremely akward proof that
$$\lim_{h\to0} \frac{|\sum_{k = 1}^{+\infty}h_kh_{k+ 1}|}{\|h\|_{l_2}} = 0$$
Another way to go
In my seminar notes, it's said that $J(x) = \left<Ax, x\right>$ where $A$ is the right-shift opertor: $Ax = [0, x]$. Then there is magic: ordinary rules of matrix calculus are applied and $DJ(u) = (A + A^{*})u$. The second-order Frechet derivative is left as homework.
Questions
- Can the first solution be somehow linked with the second?
- Am I right that $D(DJ(u)) = DJ(u)$? Here is my proof (this has been shown incorrect but yet unclear to me why): $$ \lim_{h\to0} \frac{|(DJ(u))(s + h) - (DJ(u))(s) - D(DJ(u)(s))(h)|}{\|h\|_{l_2}} = \lim_{h\to0} \frac{|\left<(A + A^*)u, s + h\right> - \left<(A + A^*)u, s\right> - D(DJ(u)(s))(h)|}{\|h\|_{l_2}} = \lim_{h\to0} \frac{|\left<(A + A^{*})u, h\right> - D(DJ(u)(s))(h)|}{\|h\|_{l_2}} $$
- Why is it suddenly possible to apply matrix calculus to an operator which isn't working with finite spaces? I mean, $A$ is not just a matrix, right?
If you indeed find extremely awkward proofs tempting, then go for it... but a proof of $$\lim_{h\to0} \frac{|\sum_{k = 1}^{+\infty}h_kh_{k+ 1}|}{\|h\|_{l_2}} = 0$$ need not be awkward. Since $2h_{k}h_{k+1}\le h_k^2+h_{k+1}^2 $, the numerator is at most $\|h\|_{l_2}^2$.
In the second approach, instead of manipulating with a particular quadratic form, we work with a general one. It's like solving $x^2+5x+6$ with factorization trick versus deriving a formula for solution of $ax^2+bx+c=0$ and then just using it.
Because it's been proved to work. And if it wasn't, it should have been done. When a professor skips a proof, it's not as much magic as laziness or lack of time in lecture.
The proof isn't difficult, though: $$\left< A(x+h),x+h \right>-\left<Ax,x\right> = \left<Ax,h\right>+\left<Ah,x\right>+\left<Ah,h\right>$$ The second term on the right is $\left<A^*x,h\right>$, and the last one is bounded by $\|A\| \|h\|^2$.
No, the second derivative is never equal to the first derivative. They have different domains. The second derivative at $x$ takes two vectors as arguments, the first derivative takes only one. Correct formula: $$D^2J(u)(h,k) = \left<(A+A^*)h,k\right>$$ This is using the idea of second derivative as a bilinear form $Q$ on vectors $h,k$ such that $$J(u+h+k) - J(u) - DJ(u)(h+k) = Q(h,k)+o(\|h\|^2+\|k\|^2) $$ In your case the computation is simple because the $o$ term is actually zero.
Above, "bilinear form" is a linear map from $X$ to the space of linear functionals. So, the second derivative is a map from $X$ to $L(X\to L(X,\mathbb R))$, which happens to be constant in this case (there is no $u$ in the second derivative).