I understand the definition of initial and final objects in a category: that an object $\frak{I}$ is initial in a category $\frak{C}$ if for every other object in $\frak{C}$ (we'll just call it $\frak{O}$), there is a unique morphism $\frak{I} \rightarrow \frak{O}$. Similarly for final objects with the arrow reversed - there should be a unique morphism from that object to the final object.
My question is a little bit vague. In some sense, I want to understand how knowledge of the initial and final objects in a set help you unravel the content of the category in some way. For example, In $\frak{Set}$, the only initial object is $\emptyset$, and the final objects are all the singletons. On the other hand, in $\frak{Grp}$, the trivial group $\{e\}$ is both initial $and$ final.
The proof of these claims is pretty trivial. However, I expect that this should help me to understand exactly what makes these categories (and their objects) different. In this case, the difference arises from the requirement that the morphisms in $\frak{Grp}$ must preserve identities. How can I go about understanding these differences in other categories? What is the general principle at work when parsing language involving initial and final objects in general? That is to say, what mathematical content do these objects contain?
Zhen Lin is correct that you almost always care about initial/final objects in related categories rather than (or rather, in addition to) the main category of interest. This isn't too surprising since a category can have at most one initial object up to unique isomorphism. These related categories are almost always comma categories.
The significance of initial/final objects is that they are one of three ways of formulating universal properties, a (or even the) core idea in category theory. Arguably, (1-)category theory is the study of universal properties.
I recently wrote a blog post providing an overview of these three ways of formulating universal properties and how each of these perspectives relate to different styles of doing category theory. Being familiar with all three perspectives and how they relate is, of course, important.
Briefly summarizing, as mentioned above, with initial objects you tend to make companion categories to study properties of some given category. Universal arrows can be formulated as initial objects in a particular comma category. Using universal arrows you usually use diagram chasing to prove theorems. This style doesn't involve defining other categories as much but more finding the universal arrows in your category. Finally, the third perspective is representability where you work directly with hom functors and can often utilize basic set theoretical tools.