If we have a topological space $X$ and a subset $A \subset X$, we can define $X \backslash A$. My question is: Is it true that this equivalence relation is closed iff $A$ is closed as a subset of $X$ and similarly for open sets $A$? This sounds natural, but I don't know whether it is true.
Forthermore, we can define quotient spaces by using a group action. So if we have a group acting on $X$, we can define $X \backslash G$ the space of all orbits. Is there any way to deduce from the structure of the group or space whether this equivalence relation is open or closed?
I assume by a "closed equivalence relation" you mean one whose equivalence classes are all closed sets. The equivalence classes are $A$ and all the individual points of $X-A$. So if $A$ is closed and $X$ is Hausdorff then yes, the equivalence relation is closed. But the only way the equivalence relation could be open would be if $X-A$ is a discrete subset of $X$.
For your second question, the study of group actions on topological spaces is deep and multifaceted. There is no way to deduce what you want purely from the group structure: groups such as $SL(2,\mathbb{Z})$ have actions in which every orbit is discrete, actions with a dense orbit, and actions with many intermediate behaviors. There is no way to deduce what you want purely from the topology of $X$: the circle $X=S^1$ has group actions with one orbit ($S^1$ is itself a group acting on itself by left multiplication), with dense orbits (the action of the rational subgroup subgroup of $S^1$), and with discrete orbits (the action of any finite subgroup of $S^1$).
Generally speaking, the properties you ask about are delicate properties about the group $G$, the space $X$, and the action of $G$ on $X$.