Commonly, we consider for a fixed group $G$ the category $\mathcal{A}_G$ where the objects are group actions $G \times X \to X$ and a morphism between two group actions $\alpha: G \times X \to X$, $\beta: G \times Y \to Y$ is a set function $f: X \to Y$ such that we have $\forall x \in X: f(g \cdot_{\alpha} x) = g \cdot_{\beta} f(x)$.
Now, what about the category $\mathcal{B}$ where the objects are group actions $G \times X \to X$ and a morphism between two group actions $\alpha: G \times X \to X$, $\beta: H \times Y \to Y$ is a pair $(\varphi, f)$ such that $\varphi: G \to H$ is a group homomorphism, $f: X \to Y$ is a set function and such that we have $\forall x \in X: f(g \cdot_{\alpha} x) = \varphi(g) \cdot_{\beta} f(x)$? Does this category have a common name? Where can I read more about this category?
Additionally, how do common categorical constructions look like in these categories? For example, what are the initial and terminal objects and how do products and coproducts look like?
My guess would be that in $\mathcal{A}_G$, the initial object is the trivial action $G \times \emptyset \to \emptyset$, the terminal object is the trivial action $G \times \{ * \} \to \{ * \}$, the product of $\alpha$ and $\beta$ is the natural action $G \times (X \times Y) \to X \times Y$ and the coproduct of $\alpha$ and $\beta$ is the natural action $G \times (X \sqcup Y) \to X \sqcup Y$. In $\mathcal{B}$, my guess would be that the initial object is the trivial action $\{ e \} \times \emptyset \to \emptyset$ and the terminal object is the trivial action $\{ e \} \times \{ * \} \to \{ * \}$. However, I'm very unsure about the product and coproduct in $\mathcal{B}$.
Is the construction of $\mathcal{A}_G$ related to the construction of the coslice category? Is the construction of $\mathcal{B}$ related to the construction of the comma category?
Except for the question of coproducts in $\mathcal{B}$ (which I don't think it has, but I have not a proof of that), this answer should cover your questions. Edit: $\mathcal{B}$ is complete and cocomplete, the proof of which I have edited in below.
Write $BG$ for the category on one object $*$, with hom-set $BG(*,*)=G$. This is the categorical version of the group $G$. Now, you can check that $\mathcal{A}_G\simeq\mathrm{Fun}(BG,\mathsf{Set})$ as categories, sending a set $X$ with group action by $G$ to the functor $BG\to\mathsf{Set}, *\mapsto X, g\mapsto (g\cdot -)$. In particular, we can compute limits and colimits in $\mathcal{A}_G$ objectwise after translating everything in sight to functors $BG\to\mathsf{Set}$ (as limits and colimits in functor categories are computed objectwise), so your descriptions of limits, colimits, initial and terminal objects in $\mathcal{A}_G$ are correct.
The category $\mathcal{B}$ is quite interesting actually. There is an assignment $\mathsf{Grp}^\mathrm{op}\to\mathsf{Cat}_{(2,1)}, G\mapsto \mathcal{A}_G$, where $\mathsf{Cat}_{(2,1)}$ is the $(2,1)$-category of categories. This assignment sends a group homomorphism $\varphi\colon G\to H$ to the functor $\varphi^*\colon \mathcal{A}_H\to\mathcal{A}_G$ which sends an $H$-set $X$ to the $G$-set $\varphi^*X$ with same underlying set $X$, and $G$-action given by $g\cdot x:=\varphi(g)\cdot x$. Now, you can argue if the assignment $\mathsf{Grp}^\mathrm{op}\to\mathsf{Cat}_{(2,1)}$ is actually a functor, or if it is a ''pseudo-functor'', i.e. a (weak) $(2,1)$-functor, but for my story it doesn't matter and we can just take it to be a $(2,1)$-functor. We now invoke the Grothendieck construction, which asserts that this data of a contravariant $(2,1)$-functor to $\mathsf{Cat}_{(2,1)}$ is the same data as some so-called Grothendieck fibration $U\colon\mathcal{C}\to\mathsf{Grp}$, for some category $\mathcal{C}$. Intuitively, $U$ encodes the functor $\mathsf{Grp}^\mathrm{op}\to\mathsf{Cat}_{(2,1)}$ by satisfying $U^{-1}(G)\simeq\mathcal{A}_G$, and a group homomorphism $\varphi\colon G\to H$ will induce (because $U$ is a Grothendieck fibration) a functor $U^{-1}(H)\to U^{-1}(G)$, which is up to equivalence given by $\varphi^*\colon \mathcal{A}_H\to\mathcal{A}_G$. So $\mathcal{C}$ is sort of obtained by ''gluing together'' all the categories $\mathcal{A}_G$ over varying $G$ together, making it a category that lies over $\mathsf{Grp}$.
Now, what is this fabled Grothendieck fibration in our case? It is given by $\mathcal{B}$, together with the forgetful functor $\mathcal{B}\to\mathsf{Grp}, (G,X)\mapsto G$. The proof is not that hard if you know what the Grothendieck construction is and how it works, and I will omit the proof for reasons of length. But at least this gives you a good way of understanding how $\mathcal{B}$ is related to all the categories $\mathcal{A}_G$: it is in a very precise sense obtained by ''gluing'' all these categories together along the maps $\varphi^*\colon \mathcal{A}_H\to\mathcal{A}_G$ for group homomorphisms $\varphi\colon G\to H$. Incidentally, it also tells you that $\mathcal{A}_G$ is equivalent to the fiber over $G$ of the forgetful functor $\mathcal{B}\to\mathsf{Grp}$, which is also intuitively true.
Given a $G$-set $X$ and an $H$-set $Y$, the product of $(G,X)$ and $(H,Y)$ in $\mathcal{B}$ is the $(G\times H)$-set $X\times Y$ with diagonal action. Indeed, given a $K$-set $Z$ and maps $(\varphi,f)\colon (K,Z)\to (G,X)$ and $(\psi,g)\colon (K,Z)\to (H,Y)$, the set map $(f,g)\colon Z\to X\times Y$ induced by them is equivariant in the sense that it is a morphism in $\mathcal{B}$, and this is even on the level of sets the only map that can make the necessary triangles commute.
On the other hand, the coproduct of $(G,X)$ and $(H,Y)$ is generally not the $(G*H)$-set $X\sqcup Y$, where $G*H$ is the free product of groups (also the coproduct in $\mathsf{Grp}$), with the action of $G*H$ on $X\sqcup Y$ is specified by saying that $G$ acts as usual on $X$ and acts trivially on $Y$, and similarly for $H$. I don't see what it has to be, so I am not sure if $\mathcal{B}$ even has coproducts in general.
Edit: The category $\mathcal{B}$ is cocomplete. In fact, reading the nLab page on the Grothendieck construction that I linked above as well, we can go to Proposition 3.1, which tells us that $\mathcal{B}$ is cocomplete when the following three conditions are satisfied:
All in all, we see that $\mathcal{B}$ is actually cocomplete. Moreover, since the functors $\varphi^*$ are also right adjoints (as $(B\varphi)^*$ admits a right adjoint via right Kan extension), the same Proposition 3.1 on the nLab page linked above also gives you that $\mathcal{B}$ is complete.
The terminal object of $\mathcal{B}$ is indeed $(\{e\}, *)$, as a map $(\varphi,f)\colon(G,X)\to (\{e\},*)$ can only consist of the unique map $\varphi\colon G\to \{e\}$ of groups, and the unique set map $X\to *$ is equivariant with respect to this. The initial object of $\mathcal{B}$ is $(\{e\},\varnothing)$, which is likewise not hard to check.