There are well-understood theorems that give sufficient conditions for a functor $R: D\to C$ to have a left adjoint. For example, $R$ should preserve limits and $D$ should have nice categorical properties.
But there are situations in which $R$ does not preserve limits, yet it lifts to a right adjoint functor $R: D\to C'$, where $C'\to C$ is some nice projection of categories.
For example, if $R$ is the forgetful functor from the category of fields to the category of rings, then $R$ does not preserve products, but we can define $C'$ to be the category of pairs $(R,p)$, where $R$ is a ring and $p$ is a prime ideal of $R$. Then the functor $R':D\to C'$ sending $k$ to $(k,(0))$ has a left adjoint sending $(R,p)$ to $Frac(R/p)$.
This has a natural description in terms of comma categories $(c/R)$, though I am not sure if this is fruitful to pursue.
I believe that this is a very general phenomenon, but I am short on both examples and theory. Are there results or general constructions that begin to describe what is happening here?
Let $R : D \to C$ be any functor. Let $C'$ be the category of triples $(c, d, f)$ where $c \in C, d \in D$, and $f : c \to R(d)$ is a morphism; this is a certain comma category, as you say. It admits a projection to $C$ given by forgetting $d$ and $f$, and it also admits both a functor to and from $D$. The functor from $D$ is given by
$$R' : D \ni d \mapsto (R(d), d, \text{id}_{R(d)}) \in C'$$
and the functor $L : C' \to D$ is given by forgetting $c$ and $f$.
Proof. We want to show that we have a natural bijection
$$\text{Hom}_D(d, d') \cong \text{Hom}_{C'}((c, d, f), (R(d'), d', \text{id}_{R(d')})).$$
This is just a matter of unwinding the definition of a morphism in $C'$. A morphism from $(c, d, f)$ to $(R(d'), d', \text{id}_{R(d')})$ consists of a morphism $g : c \to R(d')$ and a morphism $h : d \to d'$ such that
$$\left( c \xrightarrow{g} R(d') \right) = \left( c \xrightarrow{f} R(d) \xrightarrow{R(h)} R(d') \right).$$
This uniquely determines $g$ given $h$, and there is no condition on $h$. So we get a natural bijection to morphisms $d \to d'$ as desired. $\Box$
One way to think about this construction is to think about comma categories as a lax variant of homotopy pullbacks of spaces. Taking comma categories where one of the functors is the identity corresponds to taking homotopy pullbacks $D \times_C C'$ where $C' \to C$ is a homotopy equivalence, and in this case the projection to $D$ is also a homotopy equivalence. And it's natural to think of adjunctions as a lax variant of homotopy equivalences; for example, they induce homotopy equivalences on nerves.