The following question is part of a homework exercise on portfolio theory that I have to do.
Suppose that $Y$ is a random variable representing the returns on an investment. Now, let $f$ be a continuous, concave, strictly increasing function of $Y$. Let $\xi^*$ be the amount invested in security Y that maximizes $E[f(\xi^*Y)]$. Show that $(\xi^*>0)$ iff $E[Y]>0$.
The function $u$ can be seen as a utility function, hence the idea is to find where your expected marginal utility is 0, similar to normal economics with deterministic payoff functions and utilities. Since the function is integrable over the reals, you can take the derivative inside the integral of the LHS of your objective function to get the marginal expected utility function: $\frac{d}{d\xi}\int u(\xi y) f(y) dy = \int \frac{d}{d\xi}u(\xi y) f(y) dy = \int yu'(\xi y) f(y) dy = E[Yu'(\xi Y)] =$ expected marginal utility.
Setting it to zero gives you the condition you mentioned: $E[Yu'(\xi^* Y)] =0$
Now, using Jensen's inequality, we know $E[u(\xi Y)] \leq u(\xi E[Y])$ since the function is strictly increasing, convex function. Therefore, we can only hope to have a positive utility on our investment if $E[Y]>0$. There is an implicit assumption that I made here that $(x<0) \longrightarrow (u(x)<0)$, otherwise, you would benefit from a loss.