Paolo Ruffini famously wrote a work providing the first proof of the unsolvability of the quintic with the extraordinary title "Teoria Generale delle Equazioni, in cui si dimostra impossibile la soluzione algebraica delle equazioni generali di grado superiore al quarto".
Are there English or French translations of this book? Online is preferable, paper is better than nothing. I'll also accept commentaries rather than complete translations, as long as they go into detail on what exactly was proved in it and how Ruffini proved those things. I'm interested in how Ruffini proved group-theoretic results without a general notion of group.
Two useful references, both available online: Niels Hendrik Abel and Equations of the Fifth Degree, by Michael Rosen: the 4th section discusses Ruffini's proof, and Niels Henrik Abel and the Theory of Equations, by Henrik Kragh Sørensen: the fifth chapter is about Ruffini's and Cauchy's work. The discussions are pretty extensive, enough to enable you to get a good of idea of Ruffini's reasoning.
The quick answer to your question about "group theory without groups" is that the discussion was phrased in terms of permutations of the roots. This idea goes back to a famous paper by Lagrange, Réflexions sur la résolution algébrique des équations, where he analyzed the known solutions for the quadratic, cubic, and quartic, in terms of ambiguities: the $\pm$ in the quadratic formula, the three-fold choice of cube roots in the cubic formula, etc. You can see the germs of the automorphism concept here.
Not only was the early work "group theory without groups", it was also "field theory without fields". Instead of working with our notion of a field, they worked with the concept of a "rational expression". While we would say something like, "the roots belong to the field generated by such-and-such", they would say that the roots were rationally expressible in terms of such-and-such. This point of view persisted until the late 19th and early 20th century, when algebra was recast in set-theoretic terms thanks to the work of Dedekind, Hilbert, Noether, and others.