Say I have a $512\times 1$ vector, is it possibly to project this vector to lower dimensions while still retaining some approximation of the data?
Not sure if this question make sense.
Basically I have some set of $512\times 1$ vectors, these vectors represent an embedding of sentences from a machine learning model. I would like to visualize these embedding in 3D space.
I have two ideas for this:
1.) Somehow reduce to $512\times 1$ vector to a $3\times 1$ presentation. I know Principle Component Analysis can do this for a $N\times M$ matrix, but can I do it on a $N\times 1$ vector.
2.) Create a visualization with variable axes. So kind of slicing the $512$ space up into many different $3$D slices.
Do either of these approaches make sense mathematically or are they nonsense?
It sounds like what you're looking for is something along the lines of the "eigenfaces" application of PCA.
I recommend you apply PCA to some set of "training data", which consists of $M$ column-vectors of length $N$. Let's say that $A$ is the training data matrix, and $\mu$ is the "average" matrix, i.e. $\mu = \frac 1{M} Aee^T$ where $e = (1,\dots,1)^T$. Let $u_1,u_2,u_3$ denote the first $3$ principal components, i.e. the first $3$ columns of $U$ in the SVD $(M - \mu) = U\Sigma V^T$.
For your new measurement, a single $N \times 1$ vector $v$, the $3 \times 1$ vector $(u_1^T v, u_2^Tv,u_3^Tv)$ can be thought of as your projection that (in some sense) retains information about the data; moreso than any other projection onto $\Bbb R^3$.
For more information on that, I would recommend looking at Bishop's "Pattern Recognition and Machine Learning".