Given $x,y \in \mathbb{R}^n,$ the idea is that the star product of $x$ and $y$ is the set of all possible dot-products of $x$ and $y$ as we permute one of these vectors while holding the other steady, with the added feature that each real number obtained in this way gets indexed by the permutation that produced it. Formally:
Definition. Given $x,y \in \mathbb{R}^n$, define that the star-product of $x$ and $y,$ written $x \star y$, is the formal $\mathbb{R}$-linear combination of permutations of $n$ given as follows. $$x \star y = \sum_{\pi \in S_n}(x \bullet (y \circ \pi)) \cdot \pi$$
This defines a bilinear map $$\star : \mathbb{R}^n,\mathbb{R}^n \rightarrow \mathbb{R}[S_n]$$
For example:
$$(1,2) \star (3,4) = (1,2) \bullet (3,4) \pi_{(0)(1)}+(1,2) \bullet (4,3) \pi_{(0,1)} = 11\pi_{(0)(1)}+10\pi_{(0,1)}$$
My motivation for considering this thing is that the rearrangement inequality gives upper and lower bounds on the coefficients.
Question. What is the "star product" of vectors really called, and where can I learn more about it?
As I see it, you have $\mathbb{R}^{n}$ along with a group action of $S_{n}$ acting by permuting coefficients of your vectors (wrt lets say the standard basis). For each permutation $\pi$, you have a map $\varphi_{\pi}:\mathbb{R}^{n} \times \mathbb{R}^{n} \to \mathbb{R}$ by $\varphi_{\pi}:(x,y) \mapsto x \cdot y^{\pi}$ (each of these is actually a nondegenerate bilinear form).
Now you want to associate each $(x,y) \in \mathbb{R}^{n} \times \mathbb{R}^{n}$ with an element of the regular $\mathbb{R}S_{n}$-module (or considered as a group algebra, depending on what you want to do with it) by $$\star : (x,y) \mapsto \sum_{\pi \in S_{n}} \varphi_{\pi}(x,y)\pi.$$
For your actual question: "what is this map actually called?" Quite possibly, whatever you want to call it. I don't entirely know your motivation for looking at it; the fact that the coefficients are (sharply) bounded by the rearrangement inequality is trivial from the definition, as is the fact that there is a natural action of $S_{n}$ on $\mathbb{R}S_{n}$ fixing each element in the image of $\star$. It is quite possible that there are some other interesting properties hiding in there, it could be fascinating to look at the geometry of this image set (though I'm guessing it is unlikely it has been considered before).