Let $\mathcal{C}$ be a category and denote $\hat{\mathcal{C}}$ for the category of presheaves on $\mathcal{C}$. For $k: K \to F$, denote $k^*$ for the pullback functor $\hat{\mathcal{C}}/F \to \hat{\mathcal{C}}/K$. Denote $\Sigma_k$ for the postcomposition by $k$ as a functor $\hat{\mathcal{C}}/K \to \hat{\mathcal{C}}/F$ and note that $\Sigma_k \dashv k^*$.
I want to find exponents in $\hat{\mathcal{C}}/F$. For some $g: G \to F$, $h: H \to F$, $k: K \to F$, we must have that
$$\begin{align} \hat{\mathcal{C}}/F(k, h^g) &\simeq \hat{\mathcal{C}}/F(k \times g, h)\\ & \simeq \hat{\mathcal{C}}/F(\Sigma_k \circ k^*(g), h) \\ &\simeq \hat{\mathcal{C}}/K(k^*(g), k^*(h)). \end{align}$$
For $k$ the identity on $F$, this says that the sections of $h^g$ are in natural bijective correspondence with $\mathcal{C}/F(g, h)$ which is neat I suppose, but does not bring us a whole lot further.
I want to take $k$ a morphism between the Yoneda embedding $y_C$ and $F$ (aka an element of $FC$) for a bunch of $C$'s and try to use the Yoneda lemma and the fact that morphisms in $\hat{\mathcal{C}}/F$ are morphisms in $\hat{\mathcal{C}}$ in particular, but have not managed it, and for $C$'s such that $FC = \emptyset$, they do not even exist.
Let me first treat the case of the topos of sets, and then generalize.
In this case, if $F$ is a set and $x : X \to F$ is an object of $\mathbf{Set}/F$, then this object is uniquely determined up to isomorphism by the collection of sets $X_f := x^{-1}(\{ f \})$ for $f \in F$ (and in fact $X \simeq \bigsqcup_{f \in F} X_f$). Also, the functor $X \mapsto X_f$ is represented by the object $1^f := (\{ 0 \}, 0 \mapsto f)$. Therefore, if we let $[{-}, {-}]_F$ represent the exponential in $\mathbf{Set}/F$ and $[{-}, {-}]$ the exponential in $\mathbf{Set}$, then we must have: $$([X, Y]_F)_f \simeq \operatorname{Hom}_F(1^f, [X, Y]_F) \simeq \operatorname{Hom}_F(1^f \times X, Y) \simeq [X_f, Y_f].$$ Thus, $$[X, Y]_F \simeq \bigsqcup_{f\in F} [X_f, Y_f]$$ with the map to $F$ being the projection $(f, \phi : X_f \to Y_f) \mapsto f$.
It should now be straightforward to check that this indeed gives an exponential object in $\mathbf{Set}/F$.
Now, in the general case, suppose we have $F, X, Y$ objects of $\hat{\mathcal{C}}$; and denote $[{-}, {-}]_F$ for the exponential in $\hat{\mathcal{C}}/F$ and $[{-}, {-}]$ for the exponential in $\hat{\mathcal{C}}$. Then for each $U$ an object of $\mathcal{C}$, we must have $$[X, Y]_F(U) \simeq \bigsqcup_{f \in F(U)} \{ \phi \in [X, Y]_F(U) \mid \pi(U)(\phi) = f \}$$ where $\pi : [X, Y]_F \to F$ is the morphism part of the object. Now, each $f \in F(U)$ corresponds to a morphism $\tilde f \in \operatorname{Hom}_{\hat{\mathcal{C}}}(y_U, F)$, and the functor $(X, x) \mapsto X_f := \{ \phi\in X(U) \mid x(U)(\phi) = f \}$ is represented by the object $(y_U, \tilde f)$. Thus, we must have: $$([X, Y]_F)_f \simeq \operatorname{Hom}_{\hat{\mathcal{C}}/F}((y_U, \tilde f), [X, Y]_F) \simeq \operatorname{Hom}_{\hat{\mathcal{C}}/F}((y_U, \tilde f) \times_F X, Y).$$ So in summary, $[X, Y]_F$ must be isomorphic to the presheaf $U \mapsto \bigsqcup_{f\in F(U)} \operatorname{Hom}_{\hat{\mathcal{C}}/F}((y_U, \tilde f) \times_F X, Y)$ with pullback morphisms being left to the reader to fill in, and the morphism to $F$ being given at each object $U$ by the projection sending each element of this disjoint union to the corresponding $f \in F(U)$ of its part. To be more explicit, an element of $[X, Y]_F(U)$ consists of a section $f \in F(U)$ along with a function which for every object $V$ of $\mathcal{C}$, every morphism $\phi \in \operatorname{Hom}_{\mathcal{C}}(V, U)$, and every $x \in X(V)$ with $\pi(V)(x) = \phi^*(f)$, gives a $y \in Y(V)$ with $\pi(V)(y) = \phi^*(f)$ also.
Again, it should now be straightforward to verify that this does form an exponential object in $\hat{\mathcal{C}}/F$.
Incidentally, this point of view where an object of $\mathbf{Set}/F$ is viewed as a collection of sets indexed by $F$, and similarly an object of $\hat{\mathcal{C}}/F$ is viewed as a functorial collection of sets indexed by sections of $F$, is used in the interpretation of the internal language of a topos in order to give meaning to certain "dependently typed" statements. To give one example, the interpretation of the "dependently typed axiom of choice" $$I : Type, X : I \to Type \vdash \left[ (\forall i : I, \exists x : X(i), \top) \rightarrow \exists x : \left(\prod_{i : I} X(i)\right), \top \right] : \Omega$$ ends up being closely related to the condition that every epimorphism in some topos have a right inverse.