The question is straight forward:
Is there a standard terminology for a category whose objects are all free? (defined by the universal property)
The prime example I had in mind was the category of $\Bbbk$-vector spaces, but I think for any algebraic structure, the subcategory consisting of the free objects would be interesting to consider.
The notion of a "free object" doesn't make sense in an arbitrary category. Note that the universal property of e.g. free groups refers to maps from a set to the underlying set of a group. So you should have something like a forgetful functor to the category of sets in order to talk about "free objects".
If you're looking at a category of algebraic structures, in the sense of the category of algebras for a monad $T$, then the subcategory of free algebras is (equivalent to) the Kleisli category of $T$.