We have $n$ variables $x_n$ and one stochastic function $y$ of these variables. We assume that function $y$ depends on variables in the following way:
$y = c + \sum_{i=1}^n k_i x_i + \varepsilon_i$,
where $\varepsilon$ is a random component. For simplicity we can assume that the random contribution is given by a normal distribution.
We have a big number of observations that map arguments ($x_i$) into values $y$. Our goal is to find parameters $c$ and $k_i$. As far as I know there is a general analytical solution of this problem that is expressed through covariance matrix of arguments and functions. However, I cannot find it on line.
Does anybody know where I can find this solution (preferably with a derivation or an indication how it can be derived)?
One way to approach this is to fit your observations in the least-squares sense, i.e, you seek $c$ and $k_i$ to solve the problem $$ \min_{c,k} \ \tfrac{1}{2} \sum_{j=1}^m \left(y_j - c_j - \sum_{i=1}^n k_i x_{ij} \right)^2, $$ where $j$ indexes your observations and typically $m > n+1$. If you define $y = (y_1, \ldots, y_m)$ and $$ A := \begin{bmatrix} 1 & x_{11} & x_{21} & \cdots & x_{n1} \\ 1 & x_{12} & x_{22} & \cdots & x_{n2} \\ \vdots & \vdots & \vdots & & \vdots \\ 1 & x_{1m} & x_{2m} & \cdots & x_{nm} \end{bmatrix} $$ then the problem may be rewritten $$ \min_z \ \tfrac{1}{2} \|y - Az\|^2 $$ where $z := (c, k_1, \ldots, k_n)$ is the vector of parameters. The optimality conditions of this problem are $$ A^T A z = A^T y. $$ They are called the normal equations. This is a symmetric and positive semi-definite linear system. It can be solved by several different means depending on the features of your problem. If $n$ is not large, you may be able to perform a Cholesky factorization of $A^T A$. This Cholesky factorization of $A^T A$ may also be obtained by way of a QR factorization of $A$ because if $A = QR$, then $A^T A = R^T Q^T Q R = R^T R$. Or you could use the conjugate gradient algorithm to solve it. The best numerical implementations of iterative methods can be found on Mike Saunders' website: LSQR and LSMR. There are numerous other ways. Please ask if you'd like to know more.