Given a real valued set function $F$, which is defined on a class of subsets $\Sigma \subseteq \mathcal{P}(\Omega)$. ($\mathcal{P}(\Omega)$ is the power set of the universal set) Is is possible to "differentiate" $F$ somehow so that one can find its maximum and minimum values?
An application of this could be the following. If one perform a hypothesis test on $n$ samples, then we want to find an acceptance region(although it is an interval most of the time, it actually can be a general set) in $\mathbb{R}^n$ to minimize the probability of both type I and II error, $\alpha$ and $\beta$. (For example, to minimize the value of $\alpha+\beta$, a set function)
In order for differentiation to make sense, we would have to have some notion of size for the sets, which can scale arbitrarily small as we look at the difference...
Hold on, this feels familiar - let's talk measures. And there's a theorem in the Lebesgue theory relating to the "derivative of the integral".
So then, there are some circumstances in which we can have a "derivative" for $F$. We'll need a notion of measure on the underlying space - we're working with probability spaces, check. With that in mind, but also looking for a broader class of functions to differentiate, here's a prospective definition:
The derivative $F'(S)$ of $F$ at a measurable set $S$ is a function $\Omega\to \mathbb{R}$ such that $$\int_{\Omega\setminus S}F'\cdot \mathbf{1}_X\,d\mu = F(S\cup X)-F(S)+o(\mu(X))$$ for $X$ disjoint from $S$ ($\mathbf{1}_X$ is the indicator function for the set $X$), and $$\int_S F'\cdot \mathbf{1}_X\,d\mu = F(S)-F(S\setminus X)+o(\mu(X))$$ for $X$ contained in $S$.
Under this definition, what would a maximum or minimum of $F$ look like? If $F$ is differentiable and has a maximum at $S$, then $F'(S)$ is nonnegative a.e on $S$ and nonpositive a.e. on $\Omega\setminus S$. If $F$ is differentiable and has a minimum at $S$, then $F'(S)$ is nonnegative a.e on $\Omega\setminus S$ and nonpositive a.e on $S$. We're used to looking for extrema where the derivative is zero - in "nice" examples here, that will be $F'=0$ on the boundary of $S$. Of course, even stating that requires some extra structure, like a topology on $\Omega$ that our functions play nice with.