Is there any similar definition to "convex combination of vectors", for the case of matrix coefficient not the scalars?
E.g.,
$${\bf w} = {\bf (I-A) v} + {\bf A u }$$
What conditions does $\bf A$ need to satisfy to make it a "convex combination"? Is there any book introducing such topic?
Not one that I know of. Usually, the notion of a convex set is defined as follows: Let $V$ be a vector space over the field $\mathbb{R}$. Then $X\subset V$ is convex (in $V$) if for all $x,y \in X$ and for all $\theta \in (0,1)$, $\theta x + (1-\theta)y \in V$.
People usually introduce machinery when they are interested in solving a particular problem. Is your question linked to a particular problem that you can share with us?