Derivative of Var($\alpha X + (1 - \alpha)Y$) w.r.t. $\alpha$?

Question

It's supposed to result as

$ 2\alpha \sigma_x^2 - 2\sigma_Y^2 + 2\alpha\sigma_Y^2 + 2\sigma_{XY}-4\alpha\sigma_{XY} $

But I don't remember the rules of Var() algebra.

Answer 1

Variances have some unusual rules. A sort of "quasi" linear operator, if you will. Its not exactly linear over sums, and it isnt exactly linear over scalar multiples. But close. Technically it's called a quadratic form.

With regards to scalar multiples, $\mathrm{Var}(sX) = s^2 \mathrm{Var}(X)$, the scalar gets pulled out but it gets squared.

A similar thing happens with covariances, since variances are simply covariances of distributions with themselves: $\mathrm{Var}(X) = \mathrm{Cov}(X,X)$. We can observe this consistent property with scalar factors: $\mathrm{Cov}(sX, tY) = st \mathrm{Cov}(X,Y)$. Covariances are called bilinear forms, because they are independently linear in each of their arguments. Thus, $\mathrm{Var}(sX) = \mathrm{Cov}(sX,sX)= s^2\mathrm{Cov}(X,X)= s^2\mathrm{Var}(X)$.

With regard to sums, we would like it if $\mathrm{Var}(X+Y) = \mathrm{Var}(X)+\mathrm{Var}(Y)$ but this is only true for uncorrelated/independent distributions. In the general case, however, $\mathrm{Var}(X+Y) = \mathrm{Var}(X) + \mathrm{Var}(Y) + 2\mathrm{Cov}(X,Y)$. Covariance is naturally zero if and only if the distributions are uncorrelated.

Thus, for your expression, we have: $$\mathrm{Var}(\alpha X + (1-\alpha)Y) =\\ \mathrm{Var}(\alpha X) + \mathrm{Var}((1-\alpha)Y) +2\mathrm{Cov}(\alpha X,(1-\alpha)Y) =\\ \alpha^2 \mathrm{Var}(X) + (1-\alpha)^2\mathrm{Var}(Y) + 2\alpha(1-\alpha)\mathrm{Cov}(X,Y)$$

From there you differentiate with respect to $\alpha$. The traditional product and chain rules apply, and recognizing that with respect to $\alpha$, everything else is a relative constant, to be treated as such in the rules of calculus. $$\frac{d}{d\alpha}\left[ \alpha^2 \mathrm{Var}(X) + (1-\alpha)^2\mathrm{Var}(Y) + 2\alpha(1-\alpha)\mathrm{Cov}(X,Y)\right] = \\ 2\alpha \mathrm{Var}(X) - 2(1-\alpha)\mathrm{Var}(Y) + 2(1-\alpha)\mathrm{Cov}(X,Y) - 2\alpha\mathrm{Cov}(X,Y) = \\ 2\alpha \mathrm{Var}(X) -2\mathrm{Var}(Y) + 2\alpha\mathrm{Var}(Y) + 2\mathrm{Cov}(X,Y) - 4\alpha\mathrm{Cov}(X,Y)$$

Variances and covariances can alternatively be written: $$2\alpha \sigma_X^2 -2\sigma_Y^2 + 2\alpha\sigma^2_Y + 2\sigma_{X,Y} - 4\alpha\sigma_{X,Y}$$

$\blacksquare$

So yeah, the expression does look right.

---

On a side note. And thanks be to a commentor for reminding me. I thought this was superfluous, but in case youre interested...

These are some useful properties too.

Covariances are called bilinear forms because they are linear in each of its arguments. Linear in the first argument:

1. $\mathrm{Cov}(sX, Y) =s\mathrm{Cov}(X, Y)$,
2. $\mathrm{Cov}(X+Z, Y) =\mathrm{Cov}(X, Y)+\mathrm{Cov}(Z, Y)$

Linear in the second argument:

3. $\mathrm{Cov}(X, tY) =t\mathrm{Cov}(X, Y)$,
4. $\mathrm{Cov}(X, Y+Z) =\mathrm{Cov}(X, Y)+\mathrm{Cov}(X, Z)$

Hence "bilinear". Clearly we get $\mathrm{Cov}(sX, tY) = st\mathrm{Cov}(X, Y)$ from these properties. Its easy to show that $\mathrm{Var}(aX) = a^2\mathrm{Var}(X)$ given that $\mathrm{Var}(X) = \mathrm{Cov}(X,X)$.

The Variance is called a quadratic form. A function, such as $\mathrm{Var}(X)$, is said to be a quadratic form if:

1. $\mathrm{Var}(aX) = a^2\mathrm{Var}(X)$, which we've shown to be true, and
2. $\mathrm{Var}(X+Y) - \mathrm{Var}(X) - \mathrm{Var}(Y)$ is itself bilinear. In the case of variances, the expression reduces to $2\mathrm{Cov}(X,Y)$, which is bilinear.

The proofs for all of these properties fall out pretty readily from the expected value definitions of variance and covariance.