What exactly should I think of a category as? The usual definition of a category as a collection of objects + some morphisms between these objects seems a little vague to me.
Say that we have $\textbf{Set}$, the category of sets and functions and a category that I made up $\textbf{Set}_{id}$ which is the category where objects are sets and the morphisms are only the identity morphisms. Is the one I made up a real category and if so, how is it related to $\textbf{Set}$?
Now let's consider the category $\textbf{2}$ where there are two objects $A$ and $B$, their identity morphisms, and a single morphism $f : A \to B$. What does this $f$ morphism do to the elements of $A$?
Morphisms in categories don't do anything to elements of anything, in general. In your category $A$ might be Richard Nixon, $B$ the complex numbers, and $f$ my coffee mug. You introduce identity arrows, define a composition in the only possible way and you have a category. $f$ need not be a function or anything like a function.
The problem here may be less with the definition of a category and more with your not having worked through enough examples. You also may not have seen a precise definition of a category, so make sure to look one up. Do you know how every group becomes a category? Every partially ordered set? You should also check yourself whether your proposed category of sets and identity morphisms is actually a category. There is a short list of axioms that are easy to verify. If you know how to prove a set with a given operation is a group, you should have no trouble with this.