Take the Yang-Mills gauge theory for example. Gauge field $A$ is the pullback of the connection one-form to the base manifold. Other concepts of gauge theory also find their definition in fiber bundles, for example, pure gauge $g^{-1}dg$ is Maurer-Cartan one-form.
My question is that I lack a motivation to the connection one-form. The pullback of connection form is the gauge potential $A$ in physics, but there doesn't seem to be a direct physics counterpart for the connection form itself. So why are we going such a detour to define the fiber bundle and then connection forms but in physics everything seems to happen on the base manifold? From the physics point of view is there a necessity for the general connection forms? Can we formulate an alternative theory just on the base manifold?
To the last question: Yes, that's more or less what physicists do.
The main reason for a mathematician to introduce a connection-1-form is that it is a global object on the principal bundle in consideration, global in the sense that it is defined everywhere on your base manifold. The gauge potential in turn, is a differential form only locally on the base manifold, locally in the sense of 'on open sets over which the principal bundle is trivializable', and they indeed depend on a chosen bundle chart. So while the local expressions are more useful as they may be immediately used to express covariant derivatives on associated vector bundles, the lack the property of being geometric (aka. globally defined). Mathematicians, particularly in differential geometry, prefer the latter.
$\textbf{Edit:}$ and the dependency on a local trivialization lies in that 'pullback' you're speaking of, namely a pullback along a local section in the principal bundle. Note that the latter local sections are in 1-to-1-correspondence with local trivializations.