dimensionality reduction of linear regression model

Question

Context: My goal is to visualise in a plot (or multiple plots) how my linear plot fits the data.

So, I have a linear system which takes as inputs $\vec{x}=\begin{bmatrix} x_1 \\ x_2 \\ \vdots \\ x_m \end{bmatrix}$ and outputs $\vec{y}=\begin{bmatrix} y_1 \\ y_2 \\ \vdots \\ y_n \end{bmatrix}$ $$ \vec{x} \rightarrow [system] \rightarrow \vec{y}$$

As it is linear, my system can be described with the matrix A: $$[A]_{n\times m} \, [X]_{m\times1} = [Y]_{n\times 1}$$

With enough pairs of $[X]$ and $[Y]$ data, I am able to do a linear regression with the least squares method to estimate my model $[A]$.

For example if I have a set of $p$ measurements: $[A]_{n\times m}= [Y]_{n\times p} \, [X]_{m\times p}^+\> $with$ \> [X]^+$ being the Moore-Penrose invere and it works fine.

My question:

I would like to visualise this regression the same way we visualise it with the simples linear model of dimension 1: $a\cdot x= y$.

like this

Is it posible to do so by a dimensionnality reduction or at least a change of basis? What are the best methods and how to implement them? I know something similar is done in ML for high dimensionality data.

My approach:

I decomposed using SVD the matrix A so: $$U\,\Sigma\,V^H\,X=Y\\ \Sigma\,(V^H\,X)=(U^H\,Y)\\ \Sigma\,\mathcal{Z}=\mathcal{T}$$ as $\Sigma$ is diagonal I should be able to plot $min(n,m)$ regression lines with $p$ points around the line.

The problem: When $n=m$ this basically becomes $n$ linear simple regressions of 1 dimension with $p$ points around each regression: $$ \sigma_1 \cdot z_1 = t_1\\ \sigma_2 \cdot z_2 = t_2\\ \vdots\\ \sigma_n \cdot z_n = t_n\\$$ but when $n\ne m$ this gets confusing. This is probably not the right approach and PCA or SVD is usually performed on the data itself, not the linear model which confuses me. Can't we use the linear regression to perform dimensionality reduction?

Answer 1

What is the aim of the visualization?

Question 1: Do you expect linear dimensionality reduction to be effective for your data?

• If the answer is no, there is no point in plotting its projections as they will not deliver the whole multidimensional picture.
• If the answer is yes, why not do dimensionality reduction on $x$ first.

You could do it this way:

1. Do PCA on $\vec{x}$ first
2. Take a subset of PCA's, construct your linear fit, check how well it works
3. Repeat 2 by progressively adding more PCA's

Question 2:

Do you know that a linear model is a good model for your data?

• If the answer is yes, you don't need to plot anything. Everything you will ever want to know about the quality of your fit will be contained in the covariance matrix of the residual error which most linear fitting tools will provide you.

• If the answer is no, you might want to test if there exists a general nonlinear model that could fit your data. To do that, you could, for example, estimate the mutual information $I(X:Y) = H(X) + H(Y) - H(XY)$