Let $F : \mathcal{C} \to \mathcal{D}$ be an equivalence of categories, i.e., there is a functor $G : \mathcal{D} \to \mathcal{C}$ (called a quasi-inverse of $F$) such that $F \circ G \cong id_\mathcal{D}$ and $G \circ F \cong id_\mathcal{C}$. This means that for all $X$ in $\mathcal{D}$ there is an isomorphism $\alpha_X$ and for all $A$ in $\mathcal{D}$ there is an isomorphism $\beta_A$ such that for all morphisms $X \overset{\gamma}{\to} Y$ in $\mathcal{D}$ and $A \overset{\phi}{\to} B$ in $\mathcal{C}$ we have the following two commutative diagrams:
$$\begin{array}{ccc}F(G(X))& \overset{F(G(\gamma))}{\longrightarrow}& F(G(Y))\\ \alpha_X \downarrow \sim & &\sim \downarrow \alpha_Y\\ \ \ \ X&\underset{\gamma}{\longrightarrow}& Y \ \ \ \end{array}$$
$$\begin{array}{ccc}G(F(A))& \overset{G(F(\phi))}{\longrightarrow}& G(F(B))\\ \beta_A \downarrow \sim & &\sim \downarrow \beta_B\\ \ \ A&\underset{\phi}{\longrightarrow}& B \ \ \ \end{array}$$
A priori, $\alpha_-$ and $\beta_-$ are not related. At the beginning of page 31 of Ravi Vakil's November 18, 2017 draft of "The Rising Sea - Foundations of Algebraic Geometry", he states en passant that a quasi-inverse is unique up to a unique isomorphism. Now, from the two diagrams above, one infers that if $G'$ is another quasi-inverse of $F$ (with maps $\alpha_-'$ and $\beta_-'$ for the diagrams), then we have an isomorphism of functors between $G$ and $G'$ given by
$$ G(X) \overset{{\beta'}_{G(X)}^{-1}}{\longrightarrow} G'(F(G(X))) \overset{G'(\alpha_X)}{\longrightarrow} G'(X).$$
My question is: why is this unique, i.e., why can there be no other isomorphism of functors between $G$ and $G'$?
I am quite sure that in general functors can be isomorphic in many different ways, so this must have something to do with the above diagrams. To try and understand this, I looked at the easiest case: $G'=G$. Then clearly $id_{G(-)}$ defines an isomorphism of functors, but so does $G(\alpha_-) \circ \beta^{-1}_{G(-)}$ as we have seen above. So in this easier setting an embryo of my question would be:
Does the equality of maps $G(\alpha_X) = \beta_{G(X)}$ hold for all $X$ in $\mathcal{D}$?
I think that I might be missing some very easy argument, like seeing a quasi-inverse as a universal object of some kind, but I really cannot see through it.
It's simply not true: the isomorphism is not unique. Indeed, you could have $G=G'$, in which case the claim is that any functor which is an equivalence has no automorphisms besides the identity. This is obviously false. For instance, the identity functor $1:Ab\to Ab$ has multiplication by $-1$ as a nontrivial automorphism. Or if $A$ is any group considered as a one-object category, any element of the center of $A$ gives an automorphism of the identity functor.
You might demand for the isomorphism to be compatible with the natural transformations $\alpha$ and $\beta$, but then it need not even exist. Indeed, such a compatible isomorphism would preserve whether the $\alpha^{-1}$ and $\beta$ are the unit and counit of an adjunction between the two functors. This is true for some choices of $(G,\alpha,\beta)$ but not others. Again, you can get an easy counterexample using groups as one-object categories. Let $A$ be a group and $F=G=G'$ be the identity functor, let $\alpha$, $\alpha'$, and $\beta$ be the identity, but let $\beta'$ be some nontrivial central element of $A$.
The right way to get a unique isomorphism is to only consider adjoint equivalences, that is equivalences $(F,G,\alpha,\beta)$ for which $\alpha^{-1}$ and $\beta$ are the unit and counit of an adjunction. For any quasi-inverse $G$ of $F$, there exists a choice of $\alpha$ and $\beta$ which do form an adjoint equivalence. Then, between any two such choices of $(G,\alpha,\beta)$ there is a unique isomorphism which is compatible with the $\alpha$ and $\beta$ maps. This is just a corollary of the more general statement that the left adjoint of a functor is unique up to unique isomorphism preserving the adjunction, which is an immediate consequence of Yoneda's lemma when you view the adjunction in terms of Hom-sets. (In terms of your attempted argument, the condition $G(\alpha_-) \circ \beta^{-1}_{G(-)}=id_{G(-)}$ which you want is exactly one of the zigzag equations that the unit and counit of an adjunction are required to satisfy.)