The usual definition of an affine connection $\nabla_X Y$ requires $X,Y$ to be smooth vector fields on the ambiant differential manifold $M$, i.e. $C^\infty$. However at any point $p\in M$, $(\nabla_X Y)_p$ means the differential of $Y$ at $p$ in the direction $X_p$. So it seems enough that $Y$ is just differentiable, and we don't need any hypothesis on $X$, not even continuity.
Likewise, geodesics are required to be smooth curves, but their definition $\nabla_{\dot{\gamma}}\dot{\gamma}=0$ suggests that they are only $C^2$ (their acceleration is zero).
And regarding the parallel transport of a tangent vector $u\in T_p M$ along a curve $\gamma$, it seems that $\gamma$ just has to be piecewise differentiable : compose the parallel transports on each segment where $\gamma$ is differentiable. The speed $\dot{\gamma}$ does not need to exist at the junction points.
Are there generalized definitions of connections for these non smooth cases ?
One can define affine connections like you write, of course, but in practice we often need objects derived from affine connections, including tensorial ones like curvature and torsion, to have at least a certain degree of regularity, and so we typically need the ingredients themselves to have some regularity (roughly speaking, $1$ additional derivative for each covariant derivative used to define them).
More generally most of the definitions in differential geometry work just fine if we lower the requisite regularity of objects to $C^k$ for some $k$, but in general any computation that uses derivatives (hence many proofs) requires some minimum value of $k$ to work. For most purposes it's reasonable to learn the theory assuming that all objects are $C^\infty$ and then go back to a proof and verify that it still works when you need the result for some weaker regularity class, e.g., $C^k$.