If you study risk and what appropriate actions are for linear and quadratic loss, you'll find that the median and the mean are the 'optimal' actions.
On the one hand, it takes simple differentiation to prove that the mean minimizes the quadratic risk. On the other hand, the proof that the median minimizes the linear risk uses either the proper first fundamental theorem of calculus or some nifty identity. However, I can simply see that the median does its job; by shifting the action, the decrease of loss attained through a now smaller probability-mass and an increase of loss is gained through a bigger probability-mass, giving a net gain.
So with the proof, it's algebraically clear that the mean sets the gradient of the risk to zero, minimizing the risk. However, I cannot be content with only an algebraic perspective, since it doesn't feel visual enough for me... But there is hope that the concept of the concept of the moment might give the 'aha'-feeling. Why would someone compute the rotational inertia around the center of mass? That would be equivalent to computing the variance around the expected value. Maybe the answer to this can let me fully grasp the minimized risk.