I tried comparing sklearn.decomposition.KernelPCA with a linear kernel to sklearn.decomposition.PCA on the same data set and got different eigenvectors. My understanding is that these should be equivalent methods, which I thought should entail that the eigenvectors would be equal.
But it is mentioned here that "in PCA it doesn't matter where the eigenvectors point as long as lines created by those vectors have proper directions."
I take this to mean that there should be some sort of transformational invariance. It would help my understanding to see where in PCA this invariance, or at least non-uniqueness, shows up in PCA.
Assuming that the eigenvectors are normalized to have unit length, the non-uniqueness comes from needing to pick a basis for each eigenspace.
For example, let $\lambda$ be an eigenvalue of a matrix $A$. All the vectors $x$ for which $Ax = \lambda x$ forms a linear subspace, lets call that subspace $E$. The dimension of that subspace is the same as the multiplicity of the eigenvalue $\lambda$.
The issue is that there is a degree of freedom when choosing the basis of the subspace $E$. The choice of basis being the set of orthogonal vectors $x_1,...,x_d$ such that $\text{span}(x_1,...,x_d) = E$. Any choice of basis is going to be a valid set of eigenvectors, but numerically the actual basis a method provides you depends on the method used to find the eigenvectors, and if that method uses a random initialization then the actual basis you get will also be random.
As a concrete example, for the identity matrix you can choose any set of orthogonal vectors to get a basis for the eigenspace.