Covariance of an RV with an affine transform of itself

Question

It is often mentioned that the requirements of a covariance matrix are that it must be symmetric PSD.
We have a standard result $\text{Cov}[Ax, By] = A\text{Cov}[x, y]B^T$. In the context of the question this reduces to $A = I_{n_x \times n_x}$ and $B=KX+b$ where b can be neglected for the purposes of the covariance computation and so we get $\text{Cov}[x, Ax] = \Sigma^xK^T$
There is no guarantee that $\Sigma^x K^T$ will be symmetric (unlike $\Sigma^x$ or $\Sigma^y=K\Sigma^xK^T$) so how does this formula make sense intuitively if it doesn't fulfill the requirements of a covariance matrix?

Answer 1

To paraphrase your question.

$\mathsf {Cov}(x,Bx)=\Sigma^x B^\intercal$ . Why is this not guaranteed to be symmetric, when that is a property of covariant matrices?

It is because that is not a Covariant Matrix, rather it is a Joint Covariance Matrix. That is a different beast.

Covariance Matrices must be positive-definite symmetric square matrices by definition.

Being defined as the matrix of covariances among members of a random vector, they are the joint covariance matrix of a random vector and itself.

$\qquad{\Sigma^x {= \mathsf {Cov}(x,x)\\=\mathsf E(xx^\intercal)-\mathsf E(x)\mathsf E(x^\intercal)\\=\mathsf{Var}(x)}}$

The row-column element of this matrix is thus ${[{\Sigma}^x]}_{ij} = \mathsf {Cov}(x_i,x_j)$, while the transposed element is ${[{\Sigma}^x]}_{ji} = \mathsf {Cov}(x_j,x_i)$ , so these elements are certainly equal.

---

Joint Covariance Matrices need not be symmetric square matrices; as they are the matrix of the covariances among members of two random vector.

$\qquad{\mathbf K_{xy}~{=\mathsf{Cov}(x,y)\\=\mathsf E(xy^\intercal)-\mathsf E(x)\mathsf E(y)^\intercal}\\\mathbf K_{yx}~{=\mathsf{Cov}(y,x)\\=\mathsf E(yx^\intercal)-\mathsf E(y)\mathsf E(x)^\intercal\\={\mathbf K_{xy}}^\intercal}}$

The row-column element of this matrix is thus ${[\mathbf K_{xy}]}_{ij}=\mathsf{Cov}(x_i,y_j)$ but the transposition is ${[\mathbf K_{xy}]}_{ji}=\mathsf{Cov}(x_j,y_i)$, so these elements may not be equal.

---