mean and variance normalization of vectors

Question

I have vectors $x \in \mathbb{R}^n$ and I expect some multivariate normal distribution.

I want to normalize the vectors in such a way that $y = M (x - b)$ has mean zero ($\operatorname{E}[Y] = 0$) and variance 1 ($\operatorname{var}(Y) = 1_n$).

For $n=1$, that is basically the Standard score.

1. With $M = 1_n$ and $b = \operatorname{E}[X]$, I get it mean-normalized but not variance-normalized.

2. With $M$ being a non-negative symmetric square root of $\operatorname{var}(X)^{-1}$, we get $\operatorname{var}(Y) = 1_n$. We can choose $b$ as in case 1.

I want to estimate $M$ and $b$ based on some running/moving algorithm. For the first case, this is easy:

For every new $x$, I can use the update rule

$$b \leftarrow x \alpha + b (1 - \alpha)$$

for some $\alpha$. I could set $\alpha = 1/N$ and set $N \leftarrow N + 1$ in every iteration, starting with $N = 1$.

How would I do something similar for the second case, esp. for $M$?

I could use a similar update rule to calculate the covariance matrix, e.g.

$$\Sigma \leftarrow (x - b) (x - b)^T \alpha + \Sigma (1- \alpha) $$

but I'm not sure how well that works in practice (also, I haven't really seen that formula in such form) or if that even converges (given that it uses also the moving $b$).

Also, if I use this, I only have an estimation of $\operatorname{var}(X)$ but I still need the $M$. I probably could use another estimation to get the $M$ based on the $\Sigma$ (which one?), but I wonder if I could just get an estimation of $M$ more directly.

---

(I asked the same question also on MetaOptimize.)

Answer 1

I assume you want to avoid matrix inversion at each step? There are two possibilities:

I. Estimate inverse covariance directly

If you average the inverse of X*X^T instead of taking the inverse of the average of X*XT, then y will not be a normal random variable. Still, it can asymptotically look like one. See e.g.:

P. Shaman, "An approximate inverse for the covariance matrix of moving average and autoregressive processes" http://projecteuclid.org/download/pdf_1/euclid.aos/1176343085

II. Estimate covariance and invert with iterative method

Estimate covariance in the usual way:

Using homogeneous coordinates Z = [x0, x1, .., xn, 1] Keep a running weighted sum of S = Z * Z^T The mean Bj = Snj / Snn for j < n The covariance Cij = Sij / Snn for i, j < n

Then since Cij changes slowly over time, compute at each step the inverse covariance M = Inv(C) given a fast iterative method by remembering the prior Mprev, e.g.

F. Soleymani, "On a fast iterative method for approximate inverse of matrices" http://www.mathnet.or.kr/mathnet/thesis_file/CKMS-28-2-407-418.pdf