Can you point me to areas and methods that deals with modeling output from input data? Say we think our output depends on certain parameters, and say we have samples of this output versus time and samples of the parameters versus time, but we don't know how the two relate. More specifically, consider the following:
$ f \in \mathbb{R}^m $ and $A \in \mathbb{R}^{m \times n} $
Using least squares, we may find $ x \in \mathbb{R}^n $ such that $ \| f - Ax\|_2 $ is minimized. But what other ways are there to model this relationship, if, for example, we don't believe it is linear?
Thanks.
When you do OLS regression, you are actually assuming that your data comes from a Gaussian distribution whose mean is a linear function of covariates. Then, to estimate the coefficients of such a function, you use maximum likelihood; there are other methods of predicting the coefficients but ML is the simplest.
For data that is not (or can not be) Gaussian distributed, there are two straightforward ways to proceed.
We could just apply a transformation to the data poinys to make them have a close enough disyribution to use OLS. This isn't usually accurate.
Rather, we estimate the mean as a composition of a lunk function and a linear combination of covariates. This is called a Generalized Linear Model.
Past that, we can use techniques building on the GLM but I recommend first looking up all of the above. In my opinion, concise explanations of the theory are hard to come by, so I'll be more than happy to provide an in depth explanation when at a computer.