Say we have a function where the argument is pretty large horizontally, like $f(x^k,\lambda_k, \mu_k)$, or something.
Then you take the transpose of this gradient for whatever reason, be it for a product, like $\nabla_x^tf(x^k,\lambda_k.\mu_k) x^k$.
I want to know what is the proper notation here. I see some people use capital $T$, like $\nabla_x^Tf(x^k,\lambda_k.\mu_k) x^k$, so there is already a doubt on wheter capital or small $t$ is the correct.
But my real doubt is whether the transpose sign should be at the end or at the beginning. And if it's on the end, like $\nabla_xf(x^k,\lambda_k.\mu_k)^T x^k$, then should it have a parenthesis like $(\nabla_xf(x^k,\lambda_k.\mu_k))^Tx^k$, or is just adding it at the end like the first option ok? Because I can't help but to feel that without parenthesis it becomes a bit weird, but on the other hand, with the parenthesis then it looks like there is too much notation.
So, which one is best/more common?
There are many notations for the transpose of a matrix, including $M^t, M^{tr}, M^T, M^\mathsf{T}, {}^{tr}\!M$ (and others, I’m sure). All of them are fine, pick one (or use the one your professor uses) and stick with it.
If you apply an operation to the result of a function (like $\nabla$, in your case), it is occasionally clearer to put the operation next to the function (compare with $\sin^2 x = (\sin x)^2$) because otherwise one might need the extra parentheses you mention in your post. If no confusion can arise (for example in your case), this is fine.
However, for the transpose of the gradient in particular, there are also special ways of expressing this transpose, namely the Jacobian matrix (as mentioned in the comments), and the differential $\mathrm df$.
You can choose what seems clearest to you and your intended readers.