Given a set $S$, we get an endofunctor $X \mapsto [[X,S],S]$ on $\mathbf{Set}$. This is called the continuation monad for $S$, so I guess that means it's a monad. There's a natural map $$\eta_X : X \rightarrow [[X,S],S]$$ that assigns each $x \in X$ to the corresponding "evaluator" $\eta_X(x)$. Explicitly this is given by $$\eta_X(x) = (f \mapsto f(x)).$$
Okay, but what's the multiplication? $$\mu_X : [[[[X,S],S],S],S] \rightarrow [[X,S],S]$$
This seems very complicated; just saying the type of $\mu_X$ is difficult: "The function $\mu_X$ is a way of turning (ways of turning ways of turning ways of turning ways of turning elements of $X$ into elements of $S$ into elements of $S$ into elements of $S$ into elements of $S$) into (ways of turning elements ways of turning elements of $X$ into elements of $S$ into elements of $S$)."
Given $f\in [[[[X,S],S],S],S]$ and $g\in [X,S]$, define $\mu_X(f)(g) = f(\eta_{[X,S]}(g))$.
Why is this the right thing to do? Well, the functor $[-,S]$ is (contravariantly) self-adjoint, with the unit equal to the counit (in the opposite category), and the monad $[[-,S],S]$ arises from this adjunction. If you work out what the definition of the monad induced by an adjunction says in this case, you get the formula for the unit map given in your question and the formula for the multiplication given above.
A little searching on this subject led me to this question, where it's pointed out that Hayo Thielecke's PhD thesis "Categorical Structure of Continuation Passing Style" might a relevant reference.