As per the title heading,
I have always been using SVD on covariance matrices to find the principal component for a set of data points; but I was wondering if I were to put in another matrix, will the eigenvectors obtained necessarily be perpendicular to each other.
Additionally, SVD on covariance matrices result in perpendicular components due to the symmetry of the covariance matrix, or is there another reason?
Because the covariance matrix is real and symmetric, eigenvectors corresponding to distinct eigenvalues will be mutually orthogonal. If any eigenvalue is a repeated root of the characteristic equation, there will be a corresponding eigenspace in which it is possible to pick linearly independent eigenvectors that are not mutually orthogonal. However, the eigenspace can be normalised (e.g. using Gram-Schmidt) so that it is specified by mutually orthogonal basis vectors.