Is there a field of mathematics that considers multiplying functions in a manner analogous to matrix multiplication? For instance,
- Let $\mathbf{x}$ is an $n$-dimensional vector such that $x_i=\sin(2\pi \frac {i} {n}))$, for $i=1,\ldots, n$.
- Let $\mathbf{A}$ be an $n\times n$ matrix where $A_{i,j}=\cos(4\pi \frac {i} {n} - 2\pi \frac {j} {n})$ for each $i,j$.
- Let $\mathbf{y}=\mathbf{A}\mathbf{x}$ so that $y_i=\frac 1 2 \sin(4\pi \frac {i} {n})$.
If we imagine $n$ going to infinity, then we can define an operator, $\text{prod_fcn}$, such that $\text{prod_fcn}(x(t), A(t, s)) = y(t)$. This example is illustrated graphically below:

Obviously this is just one simple example of a curve being "warped" by a plane. I'm curious to know if this sort of functional operation is a commonly studied thing. If so, any information about this would be appreciated.
Looks as if you are missing a factor $1/n$?. But otherwise, your matrix sum corresponds to a discrete approximation of the following integral operator: $$ L \phi(y) = \int_0^1 k(y,x) \phi(x) \; dx $$ where $\phi(x)=\sin(2\pi x)$ and $k(y,x) = \cos(2\pi(2y-x))$. An operator of the above form and with continuous (even better $C^1$) kernel $k(y,x)$ has very nice properties, in particular may be well approximated by the discrete matrix version as in your example.
More general versions are in $L^2$ where you are dealing with Hilbert-Schmidt integral operators. But you have to be more careful when approximating by a matrix.
One may construct Fredholm determinant for such operators (which in the continuous case, you may easily approximate by your matrix version). This enables to calculate e.g. eigenvalues for the operator.
You may also have a look at Integral transforms.
These are just some examples of the use of the limiting case of your matrix multiplication.