The value of $x$ that minimises $\mathbb{E}[|X - x|]$ is the median and the value of $x$ that minimises $\mathbb{E}[|X-x|^2]$ is the mean:
$\frac{d}{dx}\mathbb{E}[|X-x|^2] = \mathbb{E}[\frac{d}{dx}|X-x|^2] = \mathbb{E}[2(X-x)] = 0$
$\Rightarrow x = \mathbb{E}[x] = \mathbb{E}[X]$
How would I go about minimising $\mathbb{E}[|X - x|^3]$ or $\mathbb{E}[|X - x|^4]$...? Is there a closed form involving different moments of $X$ for each norm? Are these measures of central tendency useful or advantageous in any way, or are they too affected by outliers?
I dont't think so for $E|X-a|^3$. But there is for the 4th moment since you may simply omit abs value. For $x\mapsto E(X-x)^4$ You'll get a forth degree polynomial involving moments up to order 4, whence need to solve a degree 3 equation for the minimum. Luckily enough There is a unique mimimum because the function is convex (why?). Useful, don't know, perhaps?