My lecture notes mention the following:
For $f : U \to \mathbb{R}$ differentiable, consider the level set
$N_y=\{x \in U : f(x)=y\}$.
Suppose that $c : I \to N_y \subset U$ is a differentiable curve. Then $f \circ c = y$ and so
$0=\frac{d}{dt}(f \circ c)=\langle grad f(c(t)), c'(t)\rangle \iff grad f(c(t) \perp c'(t)$ where the first equality follows by the chain rule.
Since this holds for any differentiable curve running through $N_y$, we can say that the gradient vector of $f$ is perpendicular to the level sets $N_y$.
The definition of the gradient used is:
Definition:
The gradient of $f : U \to \mathbb{R}$ is the uniquely determined map
$grad f : U \to \mathbb{R}^n$ with $\langle grad f(x),v=df_x(v)$ for all $v \in \mathbb{R}^n$ where $df$ denotes the differential of $f$.
If we let $\langle \cdot \rangle$ be the standar scalar product, then $grad f(x) = J_f(x)$ where $J_f(x)$ denotes the Jacobian.
Now my question is the following: I can follow all the steps in the proof, but I cannot understand the conclusion. Why does it follow that $grad f(x)$ is perpendicular to the level set $N_y$ for any $x$ and any $y$? What does it even mean to be perpendicular to a set? Does it mean perpendicular to any vector in that set?
The notes also give an example:
For $f(x)=\|x\|^2$ each vector $grad f(x)=(2x_1,...,2x_n)^T$ is perpendicular to the sphere
$N_{\|x\|^2}=\{p \in \mathbb{R}^n : \|p\|^2 = \|x\|^2 \}$.
I have tried the following to see if my interpretation of perpendicular to a set is correct:
Let $x=(1,...,1)^T$, then $\|x\|^2=n$. Now consider a vector $y$ in $N_{\|x\|^2}$, say $y=(\sqrt{n},0,...0)^T$, then $\|y\|^2=n$ (x and y have the same length). But $\langle x,y \rangle = \sqrt{n} \neq 0$. So $x$ and $y$ are not perpendicular. So I think my interpretation of perpendicularity to a set must be wrong.
Please tell me what I am missing here. How does the conclusion follow and how to interpret it? I know there are a lot of questions on this topic, but none of them could clarify this point in a satisfactory way for me.
Thank you very much!
Edit:
Maybe I should add another thought. The proof shows that $grad f(c(t)$ is perpendicular to $c'(t)$ which is the tangent vector of $c$ at $t$. So we need to show that the tangent vector points in the same direction as the vectors in the level set.
The reasoning is as follows. You start out assuming that the level curve, $\left\{(x,y) : f(x,y) = c\right\}$, can be parameterized by some function $\mathbf{r}(t), t \in [a,b]$. This turns out to follow from something called the Implicit Function Theorem. Then for each $t \in [a,b]$, $(f\ \circ \mathbf{r})(t) = f(\mathbf{r}(t)) = f(x,y) = c$. In particular, $$(f\ \circ \mathbf{r})(t) = c $$
Differentiating both sides with respect to $t$ and applying the chain-rule gives:$$0 = (f\ \circ \mathbf{r})'(t) = \nabla f(\mathbf{r}(t)) \ \cdot \mathbf{r}'(t) $$
Remember that for each $t$, $\mathbf{r}(t)$ corresponds to some point on the level curve, and that therefore $\mathbf{r}'(t)$ is tangent to the level curve at each point. Recall that the dot product being zero means that the two vectors are at right angles with each other. Therefore at each point, the gradient is at a right angle with a vector tangent to the curve at that point ($\mathbf{r}'(t))$ and must therefore be normal to the curve.
EDIT-- IN RESPONSE TO YOUR COMMENT
Let's do the circle instead of the sphere, say of radius 3. $$f(x) = \Vert x \Vert ^2 = x^2 +y^2 = 9$$
The graph of that is your level curve. We can parameterize that by $$\mathbf{r}(t) = \left(3\cos(t),3\sin(t)\right), t \in [0,2\pi]$$
Note that this works out since: $$f(\mathbf{r}(t)) = f\left(3\cos(t),3\sin(t)\right) = 9\cos^2(t) + 9\sin^2(t) = 9\left(\cos^2(t) + \sin^2(t)\right) = 9$$
Next we compute the derivatives: $$\nabla f(x,y) = \left(2x,2y\right)$$ $ \\ $ $$\mathbf{r}'(t) = \left(-3\sin(t),3\cos(t)\right)$$
To do a "test", at the top of the sphere, at point $(0,3)$, corresponding to $t = \pi/2$, the first should point straight up (e.g. have no x component) and the second should point horizontally (e.g. have no y component). This is fairly routine to verify.