The composition was defined as follow:
$(a,b)$ $\in$ $(R;S)$ $\Leftrightarrow$ $\exists c | (a,c) \in R \\ $&$ (c,b) \in S$ .
Also the composition was defined as the binary product of two matrix .
$SoR$ (relation of composition)$ = R * S $ (product of matrices) where the sum is the disjunction and the product is the conjunction.
If our two relations R and S are two convex polygon
Is there a geometric interpretation of the composition of two convex polygon ? for example :
The Green polygon was the searched result .
If you want a geometric interpretation, you're going to have to look at it in three dimensions. That will make some pretty pictures, but I hope you will forgive me for not drawing them.
Ignore your figure on the left, which overlays $R$ and $S$. That's just going to confuse you, because in three dimensional representation they are going to sit on different pairs of axes.
Instead, let $\bar{R}\subset\mathbb{R}^3$ be the prism obtained by placing $R$ on the $(a,c)$ plane $b=0$, and then extruding it along the $b$ direction. Similarly, let $\bar{S}\subset\mathbb{R}^3$ be the prism obtained by placing $S$ on the $(c,b)$ plane ($a=0$), and extruding it along the $a$ direction. Mathematically, you're doing this: $$\bar{R} = \{(a,b,c)~|~(a,c)\in R\}, \quad \bar{S} = \{(a,b,c)~|~(c,b)\in S\}.$$
Now you have two prisms in 3-space that will possibly intersect. Take that 3-dimensional intersection, and project it onto the $(a,b)$ axis. Mathematically, that is this: $$(R;S) = \{(a,b)~|~\exists c~(a,b,c)\in\bar{R}\cap\bar{S}\}.$$