Given a (locally finite) poset $(P,\leq)$ we can work with its incidence algebra, which is the $\mathbb{C}$-algebra with a basis element for each interval $[x,y] = \{z ~|~ x \leq z \leq y\}$. The multiplication is given by "convolution", where
$$ (\alpha \ast \beta)([x,y]) = \sum_{z \in [x,y]} \alpha([x,z]) \beta([z,y]) $$
Notice the incidence algebra is really a matrix algebra in disguise. If we look at matrices with rows and columns indexed by $P$, with usual matrix multiplication, then the subalgebra of matrices $A$ satisfying $A_{xy} = 0$ whenever $x \not \leq y$ is exactly the incidence algebra. So "convolution" is really matrix multiplication.
The other place one commonly sees convolution is on functions. Here we have two complex measurable functions $f$ and $g$ and we define
$$ (f \ast g)(x) = \int f(x) g(s-x) d \mu(s) $$
In the case of a discrete measure, this becomes
$$ (f \ast g)(n) = \sum_m f(n) g(m-n) $$
Now this, at least initially, doesn't look very much like matrix multiplication. We can give it the right number of variables by working with a 2-dimensional convolution:
$$ (f \ast \ast g)(x,y) = \sum_m \sum_n f(x,y) g(m-x,n-y) $$
This still doesn't look much like convolution of poset algebras, though.
There is a way to compute a convolution by working with matrices, using a Toeplitz matrix, but it doesn't seem to line up with the question I'm asking, and completely saturates the google results for anything to do with convolution and matrix multiplication.
Is there a way to see convolution in an incidence algebra and convolution of functions as the same thing? If not, why are they named this way? They don't even seem superficially similar to me.
Thanks in advance!
Disclaimer: this is not informed by any "historical" motivation, just some similarities in the math.
A "high-brow" version of the answer should probably be phrased in the language of https://en.wikipedia.org/wiki/Category_algebra, perhaps as follows.
Consider free $k$ vector space on the set of morphisms in a category. This has convolution product given by linear extension of composition (if morphisms are not composable, the result is zero). For discrete groups (as categories with one object and morphisms elements of $G$), this gives convolutions on group algebra (finitely supported $k$ valued functions on $G$). For locally finite posets (as categories with a morphism between $x$ and $y$ when $x\leq y$) this gives the convolution on incidence algebra. So they are the same thing.
A "mid-brow" version of the answer can also be given, as follows.
Think about convolutions on groups. That is, as per Wikipedia, on a suitable group $G$ with suitable (in particular, left invariant) measure $\lambda$ we define $(f * g)(x) = \int_G f(y) g(y^{-1}x)\,d\lambda(y)$. When $G=\mathbb{R}$ and $\lambda$ is the Lebesgue measure or $G=\mathbb{Z}$ and $\lambda$ is the the counting measure, you get the "usual" convolution. We now assume that all functions in the sequel are compactly/finitely supported.
Now there is a bijection between functions on $G$ and left invariant functions on $G\times G$ (aka functions on "intervals"). Namely, given $f:G\to k$ define $\hat{f} (x_1, x_2)=f(x_1^{-1}x_2)$. Then $\hat{f} (gx_1, gx_2)=f(x_1^{-1}g^{-1} gx_2)=\hat{f} (x_1, x_2)$) so $\hat{f}$ is left invariant. Conversely, given $\hat{f}$ we define $f(x)=\hat{f}(1, x)$. Observe that this is the same as $\hat{f}(y, z)$ as long as $yx=z$ i.e. $x=y^{-1}z$. Clearly the two operations of "hatting" and "dehatting" are inverse of each other.
Now let's rewrite the convolution formula in terms of interval functions. Using $w=xz$ and left invariance of $\lambda$
$$(\hat{f} *\hat{g})(x, y) = (f *g)(x^{-1} y) = \int_G f(z) g(z^{-1}x^{-1} y)\,d\lambda(z)=\\ \int_G f(x^{-1}w) g(w^{-1} y)\,d\lambda(w)=\int_G \hat{f}(x, w) \hat{g}(w, y)\,d\lambda(w) $$
This maybe suggests defining convolution for (not necessarily left invariant) functions of intervals by the same formula (skipping the intermediate expressions). When $\lambda$ is the counting measure, this is now a "matrix algebra" with rows and columns labeled by elements of $G$, just as in your post. Replacing $G$ with a poset (plus taking $\lambda$ to be the counting measure) and restricting to the functions non-vanishing only on pairs $(x,y)$ with $x\nleq y$ gives the poset convolution.
The two versions of the answer are "the same": consider the groupoid with elements of $G$ as objects and single morphism from $x$ to $y$ standing for multiplication by $yx^{-1}$. This is a category that has some features of a group-category (all morphisms invertible) and some of a poset-category (at most one morphism between any pair of objects). The "functions on intervals" are then precisely the "free vector space on morphisms", and the convolution of such functions is the same in both versions above.