Say we have:
(1) $J(x) = \int_{\textit{to}}^{\textit{tf}} g(x(t),\dot{x}(t),t) dt$.
We go through the general derivation and arrive at:
(2) $\delta J(x,\delta x) = \int_{\textit{to}}^{\textit{tf}} \{[\frac{\partial g } {\partial x}(x(t),\dot{x}(t),t)]\delta x(t) + [\frac{\partial g } {\partial \dot{x}}(x(t),\dot{x}(t),t)]\delta \dot{x}(t) \} dt$
The next step is applying integration by parts (and setting the endpoints of $\delta x$ to zero), which yields:
(3) $\delta J(x,\delta x) = \int_{\textit{to}}^{\textit{tf}} \{[\frac{\partial g } {\partial x}(x(t),\dot{x}(t),t)] - \frac{d} {dt} [\frac{\partial g } {\partial \dot{x}}(x(t),\dot{x}(t),t)]\}\delta x(t) dt$
The step from (2) to (3) is where I'm confused. I'm wondering why we couldn't just apply integration by parts the other way around and arrive at:
(4) $\delta J(x,\delta x) = \int_{\textit{to}}^{\textit{tf}} \{-\int [\frac{\partial g } {\partial x}(x(t),\dot{x}(t),t)]dt + [\frac{\partial g } {\partial \dot{x}}(x(t),\dot{x}(t),t)]\}\delta \dot{x}(t) dt$
Couldn't we just as easily regard $\delta \dot{x}$ as the independent variable, and then vary it arbitrarily with the condition that the endpoints of $\delta x$ are zero as well? That would mean the quantity inside the brackets of (4) would have to be zero by the fundamental lemma of the calculus of variations. I've always wondered about this because it would have stumped me. Then I read this passage in Feynman's physics lectures:
This sort of addressed part of my confusion, but I'm still not clear about it. I already know I must be wrong, but I can't really convince myself why.
It is perfectly valid to arrive at equation $(4)$ the way you have. Doing so, however, presents difficulties in applying the Fundamental Lemma of Calculus of Variations. The Fundamental Lemma says that for some function $f(x)$ which is $k$ times differentiable on some interval $[a, b]$, if we have \begin{equation} \int_{a}^{b} f(x)h(x)\,dx = 0 \end{equation} for every $h(x)$ which is $k$ times differentiable on $[a,b]$ and which has $h(a) = h(b) = 0$, we have $f(x) = 0$ on all of $[a, b]$. Doing things in the way you have proposed doesn't let us use the Fundamental Lemma of Calculus of Variations because we don't know anything about $h'(a)$ or $h'(b)$. Applying the Fundamental Lemma requires that $h'(a) = h'(b) = 0$, but we don't satisfy that condition in general.
Edit: To be more explicit (and use your notation), we know that $\delta x(a) = \delta x(b) = 0$, but we don't know anything about $\delta\dot{x}(a)$ or $\delta\dot{x}(b)$ and applying the Fundamental Theorem requires that these both be zero.
Edit 2: To see why we need $\delta x(t_o) = \delta x(t_f) = 0$, return to Step 2 of what you've done above. We have \begin{equation} \delta J(x, \delta x) = \int_{t_o}^{t_f} \left\{\left[\frac{\partial g}{\partial x}(x(t), \dot{x}(t), t)\right]\delta x(t) + \left[\frac{\partial g}{\partial \dot{x}}(x(t), \dot{x}(t), t)\right]\delta\dot{x}(t)\right\}dt. \end{equation} Now, when we do integration by parts on the second term, we end up with \begin{equation} \int_{t_o}^{t_f} \left[\frac{\partial g}{\partial \dot{x}}(x(t), \dot{x}(t), t)\right]\delta\dot{x}(t)dt = \left[\frac{\partial g}{\partial \dot{x}}\delta x(t)\right]_{t_o}^{t_f} + \int_{t_o}^{t_f} -\frac{d}{dt} \left[\frac{\partial g}{\partial \dot{x}}(x(t), \dot{x}(t), t)\right]\delta x(t)dt \end{equation} If we don't have $\delta x(t_o) = \delta x(t_f) = 0$, then the first term on the right-hand side of the above equation doesn't vanish and the proof falls apart at this step.