I have been reading a little about operads and their cousins multicategories. I am wondering what the cousins to polycategories are and why these mysterious cousins aren't popular in literature.
Also, how are symmetric monoidal categories related to polycategories?
On
As suggested in comments to another answer, the word "dioperad" is also used for a one-object (symmetric) polycategory, although the communities of people who talk about dioperads and polycategories are so disjoint that hardly anyone seems to have heard of both of them. See this question and answer.
The thing that's related to a polycategory in the same way that a monoidal category is related to a multicategory is a linearly distributive category, or perhaps a star-autonomous category depending on how much structure you ask for.
On
According to Craig Antonio Pastro's thesis, the term you are looking for is polyad:
We may describe morphisms in a one-object polycategory (or polyad) using pairs consisting of labelled cyclic graphs and their related signatures, up to renaming of nodes
(from page 19 of his thesis)
Operads are to multicategories (a.k.a. colored operads) what polycategories with one object are to polycategories. There is no special name for polycategories with one object that I'm aware of.
They encode "bialgebras" consisting of one object and multiple operations and cooperations (each with multiple inputs/outputs), whereas general polycategories encode bialgebras consisting of multiple objects and multiple operations and cooperations between them. They're not much more mysterious than polycategories.
These differ from the better-known PROPs in the shape of possible compositions (see here): suppose you have two morphisms $f : a \to (b,b)$ and $g : (b,b) \to c$, where $a$, $b$, $c$ are colors (objects). In a PROP you can plug both outputs of $f$ to the inputs of $g$ to get $g \circ f : a \to c$, wheras in a polycategory you can only plug one of the outputs of $f$ to one of the input of $g$ to get $g \circ_{i,j} f : a \to (b,c)$ (or $(c,b)$).