I am wondering can neural networks figure out some unknown transform?
I have two vectors:
$x$ - the original/truth value
$x_{t}$ - the transformed version of $x$
One way to describe it would be
$x_t=Fx$,
where $F$ is the transfer matrix (Ex. Fourier matrix in Fourier transform).
The problem here is that $F$ is an unknown transform, and it is not a regular known transform (Fourier, wavelet, ....etc.). But I have many $x$ and $x_t$, so I'm wondering if $F$ can be solved/known even empirically through neural network training?
Input of the neural network would be $x_t$, through the adjustment via weights and bias inside the network, the output should be $x$. Of course, here the transform has to be assumed the same for each transformation operation.
My vector size is 1024, meaning there are 1024 elements in both $x$ and $x_t$
Anyone can give some pointer how to do it with neural network? Seems to me a natural fit for a neural network problem. If neural network can do the job, is there any code/example/readings/literature on this topic?
I am not asking how to data fitting, that's not what I am after. I am just wondering if this problem can be done via neural network.
Thanks sincerely.
Theres quite a large literature on function approximation ability of neural networks.
I'll mention 3 papers:
Cybenko, G. (1989) "Approximations by superpositions of sigmoidal functions", Mathematics of Control, Signals, and Systems, 2 (4), 303-314
Kolmogorov, A. N. (1957). On the representation of continuous functions of many variables by superposition of continuous functions of one variable and addition. Doklady Akademii Nauk SSSR, 144, 679-681; American Mathematical Society Translation, 28, 55-59 [1963]
Telgarsky, M. (2016). Benefits of depth in neural networks, arXiv:1602.04485 [cs.LG].
The first 2 papers say that for any suitably nice function and a big enough neural net, you can approximate the function. The last paper says that with a deeper neural network, you can have more wiggles in your function and be able to represent it with the same number of nodes (in some sense). You can also see this CS Theory SE thread.
Of course, having an algorithm to achieve these function approximations from data is a different matter -- these are purely theoretical results.
You can look at some theoretical guarantees for function approximation (e.g. in mean square error as a number of samples) in
Barron, Andrew R. "Approximation and estimation bounds for artificial neural networks." Machine Learning 14.1 (1994): 115-133.
(The author has done other papers which are relevant as well)