Let $(M,g)$ be a globally hyperbolic spacetime, $S \subset M$ Cauchy hypersurface with unit normal vector field $n$, $\Sigma \subset S$ compact 2-dim. submanifold with unit normal vector field $\nu$.
I have a defined function $\exp: (- \epsilon, \epsilon) \times \sigma \rightarrow M$ defined via $\exp(r,p)=c_p(r)$ where $c_p$ is the null geodesic with $c^{'}_p(0)=n_p+\nu_p$.
Not a general definition for a critical value for a differentiable map between smooth manifolds $f:M \rightarrow N$ is $f(p)$ where $(df)_p$ is not surjective.
So a critical value of $\exp$ is a point $\exp(r,p)$, s.t. $(d \exp)_{(r,p)}$ is not surjective.
My problem is that I don't exactly know what $(d \exp)_{(r,p)}$ is supposed to be..
If I am reading your question rightly, you are looking for critical points on a geodesic.
Recall that we have geodesics defined by means of an exponential function. The remainder of this paragraph is a throwback to our Calculus 101 (or whichever the name was) days: exponential functions are of the form $\exp(f(x))$, and thus, where differentiable, their derivative shall be $f'(x) \exp(f(x))$. Observe that $\exp$ is globally positive, so critical points shall only present where $f'(x)=0$, i.e. wherever the argument $f(x)$ has a critical point. End of the throwback.
Back in our problem, singularities (critical points) will present if and only if they are provided by the spacetime.
Let's elaborate on that. First, a remark on geometry: $(d \exp)_{(r,p)}$ will fail to be surjective if and only if $c'_p(r) = 0$. Why? Recall that, in general, for any smooth function $f:M\to N$ we have that its differential will be a linear map $f:TM\to TN$ between the corresponding tangent bundles. This is just jargon for for each $p\in M$ there is a linear transformation $df:T_p M\to T_{f(p)} N$ such that vectors tangent to $M$ at $p$ are sent to vectors tangent to $N$ at $f(p)$. In this case, the geodesic $c_p(\cdot):(-\epsilon,\epsilon)\to \gamma;\quad r\mapsto c_p(r)$ shall be singular wherever its differential is not surjective, i.e. whenever its tangent vector is $0$.
Now recall that, by means of $c_p(r)$ we have a rule that associates points in our space with geodesic paths, and that these geodesic paths will evolve according to the metric $g$ we have on $M$. Wherever the metric presents a singularity, so will $c_p(r)$, which shall be the null vector there.