Certain mathematical objects can be described as a functor $F : \mathcal{J} \to \mathcal{C}$ from a small index category $\mathcal{J}$ to a bigger category $\mathcal{C}$. For example, we can think of a graph as being a functor from the category $\{V, E, s : E \to V, t : E \to V\}$ to $\textsf{Set}$.
But to describe more complex objects, we need more that one category. Instead of $\mathcal{J}$ and $\mathcal{C}$ being single categories, we could let $\mathcal{J}, \mathcal{C} : \mathcal{I} \to \textsf{Cat}$ be diagrams of categories, and then look at natural transformations $\eta : \mathcal{J} \to \mathcal{C}$.
For example, to describe binary operations, we can let $\mathcal{J}$ be a diagram with a single category $\mathcal{J}_x = \{X, Y, f : Y \to X\}$ and a functor $\mathcal{J}_f : \mathcal{J}_x \to \mathcal{J}_x$ that maps $X$ to $Y$, and let $\mathcal{C}_x = \textsf{Set}$ and $\mathcal{C}_f = (-)^2 : \textsf{Set} \to \textsf{Set}$.
Is there a name for these "generalized diagrams"?
To answer your stated question, what might we call these generalized diagrams, I would call them diagrams of shape $\mathcal{J}$ in $\mathcal{C}$ in the 2-category $\mathbf{Cat}^{\mathcal{I}}$.
In the particular case that $\newcommand\I{\mathcal{I}}\I=\mathbf{1}$ is the category with 1 object, so that $\newcommand\Cat{\mathbf{Cat}}\Cat^\I\simeq \Cat$, then $\newcommand\J{\mathcal{J}}\J$ and $\newcommand\C{\mathcal{C}}\C$ are identified with ordinary categories and natural transformations between the corresponding functors in $\Cat^\I$ are just ordinary functors from $\J$ to $\C$, which are ordinary diagrams.
We could make the following abstract definition.
Let $\newcommand\B{\mathcal{B}}\B$ be a 2-category, let $J,C\in \B$ be 0-cells (objects), then the hom category $\B(J,C)$ might be described as the category of diagrams of shape $J$ in $C$ (in the 2-category $\B$).
In the case that the 2-category is $\Cat$, this agrees with our usual definition of diagram. It also encompasses your notion of diagram, since for any 0,1, or 2-category $\mathcal{I}$, $\Cat^{\mathcal{I}}$ is also a $2$-category.