Motivation: Consider a smooth vector bundle $\pi:E\to M$, a fiber metric $g$, and an affine connection $\nabla$ compatible with $g$. Then the parallel transport $\Pi_{\gamma,t_0,t}$ determined by $\nabla$ is an isometry of the fibers of $E$.
Consider the smooth function $f:E\to \mathbb{R}$ on a vector bundle defined by $f(v):= g(v,v)$. In particular, parallel transport yields a smooth family of smooth maps $\Pi_{\gamma,t_0,t}:E_{\gamma(t_0)}\to E_{\gamma(t)}$ satisfying $$f \circ \Pi_{\gamma,t_0,t} = f,$$ for any path $\gamma:[t_0,t]\to M$.
Question: Suppose that $\pi:E\to M$ is a smooth fiber bundle and $f:E\to \mathbb{R}$ is an arbitrary smooth function. When is it possible to find, for any path $\gamma:[t_0,t]\to M$ a smooth family of smooth maps $\Pi_{\gamma,t_0,t}:E_{\gamma(t_0)}\to E_{\gamma(t)}$ satisfying $$f \circ \Pi_{\gamma,t_0,t} = f?$$
I.e., are there natural conditions that may be imposed on $f$ to ensure that this is possible?
Idea: If the level sets of $f$ are all manifolds and there exists a smooth Ehresmann connection on $E$ such that the corresponding horizontal bundle $\mathcal{H}$ is contained in the distribution $\ker df$, then I think the parallel transport determined by this connection would solve the problem. But it isn't clear to me when such an Ehresmann connection exists.
Evidently such a (linear) Ehresmann connection exists for the case of $E$ a vector bundle and $f(v) = g(v,v)$ described above.
There are many examples in which such a parallel transport does not exist (for example, take a non-constant function on the base $M$ and pull it back to $E$). However, it does exist in very specific cases, such as your example of a vector bundle with fiber metric. Let us try to generalize your example.
Let $E\to M$ be a fiber bundle with fiber $F$. So $E$ can be represented by a collection of trivializations and transition maps. The transition maps are all of the form $$\varphi_{\alpha,\beta}:U_\alpha\cap U_\beta\to \mathrm{diff}(F).$$ Let $G$ be a Lie subgroup of $\mathrm{diff}(F)$. A $G$-structure on our fiber bundle is a representation in which all the transition maps admit values in $G$.
Example: A vector bundle is merely a fiber bundle with fiber $\mathbb{R}^k$, equipped with a $GL_k(\mathbb{R})$-structure. Once you give this vector bundle a fiber metric, you actually reduce the structure group to $O_k(\mathbb{R})$.
We continue with a fiber bundle $E\to M$. To begin with, every fiber $E_p$ can be identified with the fiber model $F$ in many different ways. In fact, the set of diffeomorphisms $E_p\to F$ is diffeomorphic to $\mathrm{diff}(F)$. Once you equip $E$ with a $G$-structure, you distinguish a set of "special" identifications - those that respect the $G$-structure. The set of such identifications is diffeomorphic to $G$. (For example, if $E$ is a vector bundle, then the special identifications are the linear isomorphisms $E_p\to\mathbb{R}^k$. If you have a metric, the special identifications are the isometric linear isomorphisms).
Assume now that $E\to M$ has a $G$-structure and a compatible connection $\nabla$. That is, all the parallel transport maps of $\nabla$ respect the $G$-structure. Let $h:F\to\mathbb{R}$ be a function which respects the $G$-action on $F$. In other words, $h$ is constant on every $G$-orbit in $F$. Note that in this case, there is a well defined function $\tilde{h}:E\to\mathbb{R}$, which is induced by $h$. Namely, for $q\in E_p\subset E$ we have $$\tilde{h}(q)=h\circ\psi_p(q),$$where $\psi_p:E_p\to F$ is an identification that respects the $G$-structure. (Our assumptions guarantee that this is well defined). The conncection $\nabla$ will then satisfy what you want for the function $\tilde{h}$.
I leave it you to convince yourself that your example is a particular case of the situation described above.
Edit: Under mild further assumptions, the above story is in fact the only context in which this phenomenon occurs. Namely, let $E\to M$ be a fiber bundle equipped with a connection $\nabla$, and let $h:E\to\mathbb{R}$ be a function which is preserved by parallel transport of $\nabla$. We only add the assumption that parallel transport exists for every path (this is not always the case in general fiber bundles).
Choose a point $p\in M$, and let $G$ be the group of all diffeomorphisms $E_p\to E_p$ which respect $h|_{E_p}$. Then, construct a collection of trivializations of $E\to M$ using $E_p$ as the model fiber, where all identifications $E_{p'}\to E_p$ are obtained from parallel transport along some path. There are many different ways to do that, but all the transition maps will have values in $G$. This means that the function $h$ is of the sort described above.
Remark: Willie Wong's comment is correct and gives a very simple necessary and sufficient condition. Having said that, all the examples I have encountered of such behavior of a function and a connection originate naturally from a function on the model fiber, as described in this answer (this is definitely the case in your example). In other words, there is usually a bigger picture behind such a function.