From their work, it seems that the Ancient mathematicians were investigating a mathematical object not as a fun, but to solve some problem occurred in earlier work of someone. Lagrange, Galois, Abel considered groups for solving equation by radicals. Cauchy proved existence of subgroup of prime order, which was generalized by Sylow (and also by Frobenius in some direction). This development shows that the theorems proved by these mathematicians were related to some natural problem in Mathematics.
In such of situation of mathematical research, Frobenius obtained following result in 1901:
Let $G$ be a finite transitive permutation group on $X$ such that (1) $Stab(x)\neq 1$ and $Stab(x)\cap Stab(y)=1$ for $x\neq y$. Then the elements of $G$ having no fixed point together with identity form normal subgroup.
Question: What was the motivation of Frobenius for considering such class of groups?
Certainly he may not had though like
let $G$ be a group of permutations on $n$ letters, satisfying "this, this, this"; and let's see what happens?
In fact, one may see the modern way of choice of a problem; for example this one (just see shaded part first):
I want to know the motivation of Frobenius behind it. (As long as I have seen some papers of him, I found, he concerns some problem ocured in work of some mathematician or he generalizes results of some mathematician.)
There are two equivalent definitions of a Frobenius group: (1) a transitive non-regular permutation group in which no non-identity element fixes more than one point; and (2) a group $G$ with a nontrivial subgroup $H$ such that $H \cap H^g = 1$ for all $g \in G \setminus H$. (The $H$ in (2) is the point stabilizer (the Frobenius complement) in (1).) In some contexts in group theory, the second definition seems the more natural.
I have done a quick bit of searching, and found the following. Burnside started investigating Frobenius group two or three years before Frobenius proved the main result on the existence of a Frobenius kernel in a paper in 1901. Burnside was almost certainly trying to prove this result himself. The proof by Frobenius is an application of induced characters, which Frobenius had been studying already for some years (he had earlier proved the Frobenius Reciprocity Theorem for example), and the trivial intersection with conjugate property of the subgroup $H$ is particularly meaningful in the context of induced characters, and it enabled him to complete the proof.
So I think the answer to your question is probably that he was investigating an existing conjecture of Burnside, which he succeeded in proving.
Disclaimer: I have no particular expertise in the history of mathematics, so don't treat any of the above as gospel truth!