Let $\mathcal{C}$ be a category and let $(X_i)_{i\in I}$ be a family of objects in $\mathcal{C}$. We say that an object $X$ in $\mathcal{C}$ is the product of $(X_i)_{i\in I}$ if we have morphisms $\pi_i:X\longrightarrow X_i$ such that for every object $Y$ in $\mathcal{C}$ and a family of morphisms $f_i:Y\longrightarrow X_i$, there exists a unique $f:Y \longrightarrow X$ such that the following diagram commutes
$$\require{AMScd} \def\diaguparrow#1{\smash{\raise.6em\rlap{\scriptstyle #1} \lower.3em{\mathord{\diagup}}\raise.52em{\!\mathord{\nearrow}}}} \begin{CD} && X\\ & \diaguparrow{f} @VV\pi_iV \\ Y @>>f_i> X_i \end{CD}$$
This is how I've read we define a product in a given category $\mathcal{C}$. My question is, why do we define this universal property this way and not in any other way? Why is it interesting having this commutative diagram and not any other?
In the category of Sets, I can see how this corresponds to the cartesian product. Indeed, if we take $X=\prod X_i$ and we define $f(y)=(f_i(y))_{i\in I}$ and we take $\pi_i$ as the $i$-th projections, then we have a commutative diagram and the cartesian product becomes the product in Sets.
However, I still don't see why do we define the product in that way, it looks a bit unnatural to me. Was the definition motivated as a generalization of the cartesian product of sets so we could have a notion of it in other categories? If so, was this the only way to construct such generalization? Why is it important to have the product defined it this way?
It seems to me that by defining this object called product in any category we obtain an object with richer meaning. I wonder if there is any deep intuition about what a product is, or if it is "merely" an object that satisfies such commutative diagram and nothing else.
This definition is equivalent to requiring that there be bijections $$ \text{Hom}_\mathcal{C}(Z, X\times Y) = \text{Hom}_\mathcal{C}(Z, X) \times \text{Hom}_\mathcal{C}(Z,Y) $$ for any object $Z$ of $C$, functorially in $Z$. Here the product on the left is the mysterious "product object" of $\mathcal{C}$, which you wanted more motivation for, but the product on the right is just the usual product of sets. So it is in a sense the "natural" generalization of product in the category of sets.