Often times in estimation we are interested in minimizing the mean square error which can be thought about in a natural way. But are there ever times when we wish to minimize other error metrics such as choosing an estimator such that the probability $\ P(|X-a|>e)$ is minimized where $\ a$ is the estimate of $\ X$ and $\ e$ approaches zero (so that the probability of some small deviation is minimal).
Where can I go to learn about these alternate error metrics and how can they be thought of in a similar sense to the classic MMSE estimators?
You can start by looking at M-estimators and Robust Statistics. An example of such an estimator is the median. The median is minimizing the absolute distance instead of the squared distance.