First, does such a notion exist?
If so, would it be as trivial as the triple $(S,\alpha, \beta)$, where $(S,\alpha)$ is an F-algebra and $(S,\beta)$ is an F-coalgebra?
I'm assuming there would be more that needs to occur, such as some interaction between $\alpha$ and $\beta$, but I haven't found much information out there that I can understand.
More generally, given two endofunctors $T$ and $F$ on a category $C$, we can define a $(T,F)$-bialgebra to be an object of $C$ equipped with the structure of a $T$-algebra and an $F$-coalgebra, i.e. just a triple $(S,\alpha,\beta)$, where $\alpha\colon TS\to S$ and $\beta\colon S\to FS$.
But this is a very broad notion. As you expect, we usually want some relationship between the $T$-algebra structure and the $F$-coalgebra structure. The usual way of doing this is the notion of $\lambda$-bialgebra, where $\lambda$ is a distributive law (of $T$ over $F$): a natural transformation $\lambda\colon TF \to FT$.
A $\lambda$-bialgebra is a $(T,F)$-bialgebra $(S,\alpha,\beta)$ such that $\beta \circ \alpha = F\alpha\circ \lambda_S \circ T\beta$ (I recommend drawing the diagram for yourself).