How is iteratively reweighted least squares used for $L^p$ norm linear regression?

Question

The iterative scheme that I see everywhere in this context is $$\theta _{k+1}=\left(X^{\:t}W_k\:X\right)^{-1}\left(X^{\:t}W_k\:Y\right)$$ With the weight $W_k$ being a diagonal matrix of $$w_i=\left(y_i-\theta ^{\:t}x_i\right)^{p-2}$$ The iterative scheme is a direct result of minimizing the cost function $$J=\left(Y-X\theta \right)^tW\left(Y-X\theta \:\right)$$ I was only able to work out the derivation if by assuming that $W$ is a constant matrix, which implies $$\frac{\partial }{\partial \theta }\left(W\right)=0\:\&\:\:\frac{\partial \:}{\partial \theta \:}\left(W\theta \:\right)=W$$ However, this clearly isn't true by looking at what $w_i$ is, but the formula is somehow correct so what am I missing?

Answer 1

You can perform a reweighted linear least squares works as follows:

1. Initialize weight matrix $W$.
2. Solve linear least squares using $W$.
3. Calculate new $W$ given the solution found in 2.
4. Unless some convergence criterion is achieved, jump back to 2.

If $W$ is not used in convergence criterion, then we can do the check for convergence before calculating new $W$.

A simple linear least squares system you might want to solve :

$$\min_{\bf v}\{\|{\bf W(Mv-d)}\|_2\}$$

I trust you can do the algebra deriving the solution.