I know that there exists a connection on a principal bundle and via parallel transport it is possible to define a a covariant derivative on the associated bundle.
However, can we also define a covariant derivative on the principal bundle. I.e. something that can differentiate a section along a vector field? Or do we need a linear structure like the one in a vector bundle to 'take derivatives'?
On
It is defind in Wikipedia as $D\phi(v_0,...,v_k)=d\phi(hv_0,...,hv_k)$ where $h$ is the projection to horizontal subspace according to the given principal connection of the principal G-bundle $\pi:P\to M$.
If $\rho:G\to GL(V)$ is a representaion of $G$ on some vector space $V$, then a tensorial (or basic, i.e. G-equivariant and horizontal) k-form of type $\rho$ on $P$ can be identified with $P×_\rho V$-valued k-form on M.
As other answers/comments said, it is not a covariant derivative in the common sense, but is a covariant derivative in the sense that if $\phi$ is a $P×_\rho V$-valued k-form on M, and it is identified with tensorial form $\hat\phi$ on $P$, and $\nabla\phi$ is $P×_\rho V$-valued k+1-form on M, where $\nabla$ is the covariant derivative on $P×_\rho V$ corresonding to the given principal connection, and $\widehat{\nabla\phi}$ is the identification of $\nabla\phi$ to tensorial form on $P$, then $D\hat\phi=\widehat{\nabla\phi}$.
On
I know that there exists a connection on a principal bundle and via parallel transport it is possible to define a a covariant derivative on the associated bundle.
Ehresmann connections are the geometric version of connections. They are generally available on all fibre bundles and not just principal bundles. Let $p:E \rightarrow M$ be a fibre bundle over the base $M$. Then an Ehresmann connection is a splitting of $TE$ into 'horizontal' and 'vertical' bundles. The vertical bundle is given as $VE:=ker(Tp)$. Thus the splitting is reduced to a choice of horizontal bundle $HE$ which is complementary to the vertical bundle, ie $TE=HE\oplus VE$. This can be encoded into a projection $C:TE \rightarrow TE$ whose image is exactly $VE$ and whose kernel is $HE$. It also means that $C \in \Omega^1(M,TM)$. This is what is called a tangent bundle valued 1-form.
Now it is often stated that a connection is not covariant, for example see the many general relativity texts. And this, whilst true, is not the whole picture. The connection is not covariant when it is considered to live on the base manifold. It is, however covariant, as the construction above manifestly shows, when it is where it should be on, on the total space. When we specialise the above construction for the tangent bundle this means a connection lives on the second tangent bundle.
Secondly, parallel transport is more or less the same as an Ehresmann connection. We don't use parallel transport to 'transport' our connection from the principal bundle to an associated bundle, but through inducing it on the associated bundle. This relies on the action of the gauge structure group on a standard fibre, in this case, this will be a vector space and so a representation of the gauge structure group.
I'm using the term gauge structure group for what is called the structure group in the mathematical literature and what is known as the gauge group in the physics literature. It seems useful to have such a term to show the parallels between the two in two different languages but also to disambiguate since the gauge group is also, confusingly, used for a different but related and much larger group.
However, can we also define a covariant derivative on the principal bundle. I.e. something that can differentiate a section along a vector field? Or do we need a linear structure like the one in a vector bundle to 'take derivatives'?
Yes, you can. It is called the covariant exterior derivative. There is one on the principal bundle and also on any induced associated bundle and they are connected through a canonical isomorphism. Fix a connection $C$ as above on a principal bundle $P$ and define $C^*: \Omega(P,V) \rightarrow \Omega(P,V)$ by $(C^*\alpha)_p.(v_1,...,v_k) \rightarrow \alpha_p(Cv_1,...,Cv_k)$. Here $C^*$ is a projection onto the space of horizontal forms which is actually independent of the connection (it is the counterpart of the vertical bundle defined above, but because it is in the cotangent bundle, it is now horizontal). We now define the covariant exterior derivative:
$\nabla^C:\Omega^k(P,V) \rightarrow \Omega^{k+1}(P,V)$
by $\nabla^C:=C^* \circ d$.
Now fix a representation of $G$ on a vector space $V$. Then there is a canonical isomorphism:
$q^{\sharp}:\Omega(M,P[V] \rightarrow \Omega_h(P)^G$
We also have a covariant exterior derivative $\nabla^V_C$ induced on the associated bundle $P[V]$ and they are connected as:
$q^{\sharp} \circ \nabla_C = \nabla^V_C \circ q^{\sharp}$
The details are in section 11 of Kolar, Michor & Slovak's Natural Operations in Differential Geometry.
So from my point of view what a covariant derivative should be or should do I'd say that indeed you need a linear structure.
So in principle, when passing from simply vector space-valued functions $M\to W$ (or sections in the trivial bundle $M\times W$, resp.) to sections in a not-necessarily trivial bundle $E\to M$ with fiber-type $W$ you run into troubles defining a derivative of such functions. Usually this is paraphrased as that it is intrinsically "not possible to compare points in different fibers". However, it is possible to chose a covariant derivative, and thereby (at least locally) a frame which is "constant" or parallel to $M$ (cf. my comment above). Once you everywhere have distinguished such a notion of parallelity you can now indeed take derivatives and view it as "how a function changes compared to what we call constant", which is of course just the action of the covariant derivative. For this notion you need a vector space structure.
On the other hand, on a principal bundle you also have a notion of parallel in the sense of everywhere horizontal but as you don't have a vector space structure in the fibers, so "comparison with a constant section" works a bit different, particularly not by something having similar properties as a derivative in the above sense. Rather would you do something like @Deane in the comments above, but I'd say this is not a "derivative" in the sense of what a derivative should do.