Let $S$ be a scheme and $\tau$ some topology on $\mathrm{Sch}/S$. (I'm primary interested in \etale topology). Let $\pi:Y\to X$ be a morphism of $S$-schemes.
A twist of $\pi$ is a morphism $\pi':Y'\to X$, which becomes isomorphic to $\pi$ after some cover $\mathcal{U}$ in $\tau$. I want to understand how twists relate to torsors/non-abelian Cech-cohomology. In the literature I have only found twists of varieties, ie. the case where $S=X=\mathrm{Spec}(k)$ and $Y$ is a $k$-variety. Similarily we obtain a map (of isomorphism classes)
$$\phi: \{\text{twists of $\pi$} \} \longrightarrow \{\mathrm{Aut}(\pi)\text{-torsors} \}, $$ by associating the $\mathrm{Aut}(\pi)$-torsor $\mathrm{Iso}_X(Y,Y')$ to a twist $\pi':Y'\to X$. Is there any general criterion, where $\phi$ is bijective?
This seems like some kind of representability issue. Of course we can break this up into maps $$\phi_\mathcal{U}: \{\text{twists of $\pi$, trivial over $\mathcal{U}$} \} \longrightarrow \{\mathrm{Aut}(\pi)\text{-torsors, trivial over $\mathcal{U}$} \}, $$ for all covers $\mathcal{U}$.
In Serre's Galois cohomology, in the variety case, he assumes that $Y$ is quasi-projective.
To answer your direct question, it is useful to first abstract the situation slightly. That said, maybe the following will be helpful to help orient you to the rest of the answer.
Preamble
Since you seem to be confident with Grothendieck topologies, let me assume that you know what I mean by a site.
(NB: All of this is actually more cleanly phrased at the topos level, but that level of abstraction isn't perhaps helpful to address your specific question.)
I cannot go into too much detail in this post due to time and space constraints. Perhaps these notes of mine could be helpful if you want a more intuitive explanation (I apologize they are quite old, and perhaps not quite succinct or maximally insightful).
If you are interested in trying to pursue the material here more seriously, let me recommend some texts.
I would suggest either [Bertin] (most relevantly here being [Bertin, §1]) and [Olsson] (most specifically [Olsson, Chapter 2 -- Chapter 4]).
That said, except for maybe getting comfortable with the idea of a stack (which I can't expound on here), I hope the post is self-contained.
Torsors
Let us first be precise about what we mean by 'torsor' here. This is particularly relevant, as people sometimes (in my opinion confusingly) confuse the terms 'principal homogenous space' (i.e., a representable torsor) and 'torsor'.
So, let us fix the following data:
If $\mathcal{F}\colon \mathbf{C}^\mathrm{op}\to\mathbf{Set}$ is a sheaf of sets on $\mathbf{C}$, then by a right action of $\mathcal{G}$ on $\mathcal{F}$ I simply mean a morphism of sheaves of sets $\mathcal{F}\times\mathcal{G}\to \mathcal{F}$ such that for all objects $X$ of $\mathbf{C}$ the induced map $\mathcal{F}(X)\times\mathcal{G}(X)\to\mathcal{F}(X)$ defines a right action of the group $\mathcal{G}(X)$ on the set $\mathcal{F}(X)$.
(NB: You can of course axiomatize this at the sheaf level (without referring to the values to varying points) by using Yoneda's lemma, but I find that harder to remember precisely.)
To define torsors, it's useful to develop three extra pieces of terminology:
We then have the following easy proposition, which is left to you as an exercise.
(NB: As per usual, the way that one might think of a $\mathcal{G}$-torsor is as a 'space' with a $\mathcal{G}$-action which is 'locally trivial'.)
A morphism of $\mathcal{G}$-torsors is just a $\mathcal{G}$-equivariant morphism of sheaves (which is automatically an isomorphism). We then make the following notational definitions.
Then, as you have already implicitly pointed out one can understand the pointed sets $\mathrm{Tors}(\mathcal{G})$ and $\mathrm{Tors}_\mathcal{U}(\mathcal{G})$ (with neutral element given by the isomorphism class of the trivial $\mathcal{G}$-torsor) cohomologically.
All of this was likely already known to you, but setting our notation and terminology will make things smoother below.
Stacks
As is implicit in your question, often in nature sheaves of groups are presented to us as automorphism sheaves of some mathematical object. To make the notion of an 'automorphism sheaf' precise, and to present the generality in which one can make strong connections between 'twists' and torsors for automorphism sheaves, we need to introduce the notion of stacks.
Prestacks
To begin, let us recall that a groupoid is a category where every morphism is invertible. We denote the category of groupoids by $\mathbf{Grpd}$, where morphisms are just natural transformations.
So, in essence to every object $X$ of $\mathbf{C}$ we have a groupoid $\mathscr{M}(X)$, and for every morphism $Y\to X$ we have a natural transformation $\mathscr{M}(X)\to \mathscr{M}(Y)$.
In practice, such prestacks arrive by considering
Let us give some classic examples of prestacks in algebraic geometry. In what follows, for a scheme $S$ and $\tau\in\{\mathrm{fppf},\mathrm{\acute{e}t},\mathrm{Zariski}\}$ let $S_\tau$ denote the small $\tau$-site over $S$.
There are endless examples, but I leave it to you to consult textbooks for more.
Stacks
Now, one can think of a prestack as a 'higher category' (specifically $2$-category) version of a presheaf. There then is, as expected, the analogue of a separated presheaf/sheaf.
Some words are in order. First of all, this looks a priori quite different from the usual definition of sheaves. But, this is deceptive as the following instructive exercise shows.
In other words, this exercise shows that if $\mathcal{F}$ is just a normal presheaf as opposed to a prestack (i.e., valued in sets as opposed to groupoids) then the triple product doesn't matter, and we essentially recover the normal condition for a presheaf to be a sheaf.
So why do we need to consider triple products in the case of prestacks, and what is this funny 2-limit (as opposed to normal limit)? Both of these have to do with subtleties which occur when considering groupoids and not sets.
To start, this 2-limit is a 'stronger' notion of limit, where we not only require that the objects on the $X_i$ are isomorphic when restricted to each double product (note that 'equality' is essentially a useless notion in a non-discrete groupoid!), but that we remember this isomorphism, and the triple product is needed to ensure that these isomorphisms agree on triple overlaps.
In more concrete terms, one can think of this 2-limit as the category of $\mathscr{M}$-descent data for the cover $\mathcal{U}$ (see [Bertin, §1.2.4]) -- thus the notation $\mathbf{DD}_\mathbf{C}(\mathcal{U})$.
Let us now discuss whether the previous examples of prestacks are stacks.
Example 12 ([Olsson, Theorem 13.1.2]): The prestack $\mathscr{Ell}$ is always a stack, with no restrictions on $S$ or $\tau$.
Example 13 (see Tag 0CCH and Tag 08KE): The prestack $\mathscr{QP}$ is a stack if $S$ is the spectrum of a field, and $\tau\in\{\mathrm{\acute{e}t},\mathrm{Zariski}\}$ but often fails to be a stack in other situations (except when $\tau$ is the Zariski topology).
Example 14 (see Tag 0247): The prestack $\mathscr{QA}$ is a stack.
Example 15 (see Tag 0241): The prestack $\mathscr{S}$ is a separated prestack, but is rarely a stack.
(NB: For instance, the 'stackification' of $\mathscr{S}$ for the étale topology is (essentially) the category of algebraic spaces.)
Example 16 (see [Bertin, Proposition 2.17 and Exercise 2.18]): The prestack $\mathrm{Bun}_n$ is a stack.
Example 17 (see [Giraud, Chapitre III, §1.4.4.]): The prestack $\mathbf{Tors}_\mathcal{G}$ is a stack.
Twists
To relate stacks and torsors, we begin with a useful recharacterization of when a prestack is separated in terms of so-called Isom-presheaves.
Namely, suppose that $\mathscr{M}$ is a prestack over $\mathbf{C}$ and $A$ and $B$ are objects of the groupoid $\mathcal{M}(X)$, where $X$ is an object of $\mathbf{C}$. We then have a presheaf $$\mathrm{Isom}(A,B)\colon\mathbf{C}_{/X}^\mathrm{op}\to \mathbf{Set},$$ which associates to any $Y\to X$ the set of isomorphisms $A|_Y\to B|_Y$ in $\mathscr{M}(Y)$.
In particular, for a separated prestack $\mathscr{M}$ and an object $A$ of $\mathscr{M}(X)$ we get a group sheaf on $\mathbf{C}_{/X}$ given by $\mathrm{Aut}(A):=\mathrm{Isom}(A,A)$.
We denote the isomorphism classes (in $\mathscr{M}(X)$) of twists of $A$ by $\mathrm{Tw}(A)$ and $\mathrm{Tw}_\mathcal{U}(A)$ respectively. These are naturally pointed sets, with the neutral element being the trivial twist $A$ itself.
It is not hard to see that there are natural maps of pointed sets
$$\mathrm{Tw}(A)\to\mathrm{Tors}(\mathrm{Aut}(A)),\qquad \mathrm{Tw}_\mathcal{U}(A)\to \mathrm{Tors}_\mathcal{U}(\mathrm{Aut}(A))\qquad (\ast),$$
given by sending $B$ to $\mathrm{Isom}(A,B)$ with the natural right-action by $\mathrm{Aut}(A)$. Of course, there is no reason a priori that these are bijections. But, we have the following beautiful proposition.
So, as a corollary of this, we obtain the following.
Conclusion
To answer your original question, we see that we have given a sufficient condition for your map
$$\phi_\mathcal{U}\colon \mathrm{Tw}_\mathcal{U}(\pi\colon Y\to X)\to \check{H}^1(\mathcal{U},\mathrm{Aut}(A)),$$
to be a bijection: that $\pi$ belongs to some presubstack $\mathscr{M}\subseteq \mathscr{S}$ (with $\mathscr{S}$ as in Example 7) such that $\mathscr{M}$ is a stack on $S_\tau$.
While this doesn't provide an overly satisfying answer, it very much allows you to get a sense of what can go wrong, and gives you a way of trying to find references for the result you are looking for.
For instance, the reason that you see twists of quasi-projective varieties on the small étale site of $S$, where $S$ is the spectrum of a field, is that there the theory works well (it often goes under the heading of 'Galois descent for quasi-projective varieties)). The fact that such quasi-projective morphisms rarely form a stack in greater generality is one of the reasons you were having difficulty finding literature on it -- the theory doesn't work well there.
One last thing I'll point out is that the Corollary says that twists are (often) special cases of torsors. But, in fact, the other direction is true as well. Indeed, essentially by definition, torsors are twists of the trivial torsor in the stack $\mathbf{Tors}_\mathcal{G}$. Thus, up to technical points, I think of the theory of torsors and twists as being essentially equivalent. This is an incredibly powerful tool for thinking about both concepts.
References:
[Bertin] Bertin, J., 2013. Algebraic stacks with a view toward moduli stacks of covers. In Arithmetic and Geometry Around Galois Theory (pp. 1-148). Springer Basel.
[Giraud] Giraud, J., 2020. Cohomologie non abélienne (Vol. 179). Springer Nature.
[Olsson] Olsson, M., 2016. Algebraic spaces and stacks (Vol. 62). American Mathematical Soc..