I'm reading Mac Lane's Categories for the Working Mathematician and I'm having some trouble with exercise 5 in part IV.7.
To avoid introducing adjoint squares I will only formulate the question in the case where the $H,K,\sigma,\tau$ are all identities. This particular case should give me a glimpse of the general setting:
Given an adjunction $(F,G,\varphi)$ establish a bijection between natural transformations $FG\rightarrow Id$ and natural transformations $Id \rightarrow GF$.
I can find a bijection between natural transformations $Id\rightarrow GF$ and natural transformations $F \rightarrow F$ by factorising with the unit of the adjunctions; dually $Nat(FG,Id) \simeq Nat(G,G)$, but I do not know how to go from $Nat(F,F)$ to $Nat(G,G)$.
Thank you for your help
Let $(F, G, η, ε)$ and $(F', G', η', ε')$ be adjunctions, $F: \mathscr C → \mathscr D$, and $F' : \mathscr C' → \mathscr D'$. There is a powerful isomorphism of functors $\mathrm{Nat}(F'-, -F)$ and $$\mathrm{Nat}(-G, G'-) : [\mathscr C, \mathscr C']^{\mathrm{op}} × [\mathscr D, \mathscr D'] → \mathrm{Set},$$ and in particular, for every $K : \mathscr C → \mathscr C'$ and $L : \mathscr D → \mathscr D'$ a bijection $$\mathrm{Nat}(F'K, LF) ≅ \mathrm{Nat}(KG, G'L)$$ known as the mate correspondence. For $α : F'K ⇒ LF$, we get the mate $β : KG ⇒ G'L$ as $G'Lε ◦ G'αG ◦ η'KG$, and the inverse is defined similarly: $β : KG ⇒ G'L$ gets sent to $ε'LF ∘ F'βF ∘ F'Kη$. (This is btw. best done using pasting diagrams, which make the otherwise lengthy proof completely obvious.)
Choosing particular values for the six functors involved gives you all kinds of neat results. For example, setting $F' ⊣ G' = F ⊣ G$, $K = \mathrm{Id}$ and $L = \mathrm{Id}$, you get $\mathrm{Nat}(F, F) ≅ \mathrm{Nat}(G, G)$, which is what you wanted. Explicitly, $α : F ⇒ F$ maps to $Gε ∘ GαG ∘ ηG$.
Edit with calculated isomorphism and some other general remarks: We can get the isomorphism in a bit more systematic way (and prove two important facts while doing it). Set $F ⊣ G$ to identity adjunction and write $F$ for $F'$ and $G$ for $G'$. This gives you $\mathrm{Nat}(FK, L) ≅ \mathrm{Nat}(K, GL)$, or in other words the well-known fact that for $F ⊣ G$ we have $F∘- ⊣ G ∘ -$. Per the formula above, $α : FK ⇒ L$ gets sent to $Gα ∘ ηK$, and inversly, $β : K ⇒ GL$ to $εL ∘ Fβ$.
On the other hand, setting $F' ⊣ G'$ to identity gives us $\mathrm{Nat}(K, LF) ≅ \mathrm{Nat}(KG, L)$, or $-∘G ⊣ -∘F$. Each $α : K ⇒ LF$ gets sent to $Lε ∘ αG$, and $β : KG ⇒ L$ to $βF ∘ Kη$.
Specializing and chaining these two formulas, we get $$\mathrm{Nat}(FG, \mathrm{Id}) ≅ \mathrm{Nat}(G, G) ≅ \mathrm{Nat}(\mathrm{Id}, GF)$$ and $α : FG ⇒ \mathrm{Id}$ passes through $Gα ∘ ηG$ to $GαF ∘ ηGF ∘ η = GαF ∘ ηη$. The inverse is $β ↦ εε ∘ FβG$. Note that you can also just try and guess these formulas, and the fact that they are inverse is obvious by string or pasting diagrams.