Let $(X_1,\ldots,X_n)$ be an independent sample associated to a statistical model $(\Omega,\mathcal{F},\mathbb{P}_{\theta}).$
If we suppose that the distrubtion of each $X_i$ under $\mathbb{P}_{\theta}$ has a density function $f(\cdot,\theta)$ then we can estimative the likelihood function.
So we can perform estimation for univariate gaussian, multivariate gaussian.
Now I am interested about gaussian processes.
Suppose that we would like to estimate a covariance function $K(s,t)$ which can have any licit form.
My question is:
How can I estimate a covariance function ? What are the methods that already exist?
If the estimation is only about finite dimensional law therefor I don't understand what is the difference between estimation of multivariate Gaussian and estimation of gaussian process.
If you are using a GP model, and you have a parametric form for the covariance operator, then you can just use maximum likelihood or any other estimation approach.
However, you need to have observations associated with the covariates(observations $Y_1,...,Y_n$ associated with covariates $X_1,...,X_n$), or vice versa: if $X_1,...,X_n$ are the observations, some set of covariates $t_1,...,t_n$. Moving forward we'll use $X_i$ to denote a covariate and $Y_i$ to denote it's associated observation.
Suppose that $Y \sim \textrm{GP}(0,K_\theta(s,t))$, where $\exp(\frac{1}{2\theta}||s-t||^2)$, then you have that the marginal distributions of $Y_1,...,Y_n$ is jointly normal with distribution:
\begin{equation} \begin{split} \textrm{N}(0, K_\theta(X,X)) \end{split} \end{equation}
Where $K(X,X) = [\exp(\frac{1}{2\theta}||x_i - x_j||^2)]_{ij}$. You can optimize this over $\theta$ to get the MLE. The marginal Gaussianity gives you some asymptotic properties on the estimator for $\hat{\theta}$, but I'm not sure off the top off my head if it's the same stuff as in standard parametric settings.
You could also use some sort of empirical auto-covariance to get a nonparametric estimator of the auto-covariance operator. I don't know much about this, but I think various types exist, and you can sort of try and model different types of auto-covariances (like stationarity, whether it's isotropic or not, etc.) using different formulations. I would imagine that studying properties of these estimators is fairly technical. I couldn't find many references from a brief search.