Given an action of a group $G$ on a set $X$, one can define the stabiliser subgroup of an element $x\in X$ as $$ S_G(x):=\{g\in G:gx=x\}. $$ The stabiliser subgroup is also referred to as the isotropy subgroup in many textbooks and papers. To me, the term `stabiliser' makes more sense.
I was curious as to whether one of the terms has a higher preference in certain literature as compared to the other.
I would prefer "stabilizer" subgroup for the general concept, but "isotropy" subgroup in some specialized circumstances, particularly for a Lie group action on a manifold.
Added: You ask in the comments why "isotropy group" is used in that context. That's what I learned early in my mathematics education, although I have not traced the history. But what I can do is to look at the etymology and the current usage in science and mathematics. The words "isotropy" and "isotropic" refer to "direction", and in science that is how they have been used, although the precise usage differs from one branch of science to another, and even differs within mathematics.
In mathematics, the concept of direction does clearly exist on a manifold, for example two nonzero tangent vectors at a point $p$ have the same direction if each is a positive scalar multiple of the other. So, for example, the isotropy group at $p$ (the stabilizer group of $p$) induces an action on the sphere of directions at $p$. But the concept of direction has no meaning in general non-manifold contexts, where there are nonetheless lots of interesting group actions with interesting stabilizers.
Other sciences also use "isotropy" or "isotropic" in reference to directions, but with varying meanings. For example, originally scientists observed that the cosmic microwave background radiation is isotropic, meaning it is the same in all directions. More recently, however, scientists are finding verrrry tiny exceptions to that observation.
The cognate word "isotropic" is also used in geometry, in a slightly different fashion: an isotropic action is a Lie group acting on a Riemannian manifold so that the induced action on the unit tangent bundle is transitive. One of the earliest examples I learned as a graduate student is that the group of isometries of solv geometry --- the left invariant Riemannian metric on the 3-dimensional, solvable, non-nilpotent Lie group --- is not isotropic: there is a 1-dimensional sub-bundle of the tangent bundle that is preserved by all isometries.