This operation is similar to discrete convolution and cross-correlation, but has binomial coefficients:
$$f(n)\star g(n)=\sum_{k=0}^n \binom{n}{k}f(n-k)g(k) $$
Particularly,
$$a^n\star b^n=(a+b)^n$$
following binomial theorem.
I just wonder if there is a name for such operation and where I can read about its properties.

It's called a binomial convolution in Graham, Knuth, and Patashnik's Concrete Mathematics. I don't have that text in front of me (but I bet someone here can give you a page number), but here's a reference on the Fermat's Last Theorem blog.
It would also be worth checking out Section 2.3 of Wilf's Generatingfunctionology. This is on exponential generating functions. The property of interest is that if $F(x)$ and $G(x)$ are the exponential generating functions of $f(n)$ and $g(n)$, respectively, then $F(x)G(x)$ is the exponential generating function of $f(n) \star g(n)$.
(FYI: You can download the second edition of Generatingfunctionology from Wilf's web site.)