Setup
Let $f(x,y)$ be twice differentiable in both $x$ and $y$. Assume $\partial f/\partial x>0,\partial f/\partial y>0$ for $x,y>0$.
- $f$ is said to have increasing marginal product of input $x$ if its second derivative with respect to $x$ is positive, i.e. $\partial^2 f/\partial x^2>0$ for $x,y>0$.
- $f$ is said to have increasing returns to scale if, $\alpha f(x,y)<f(\alpha x,\alpha y)$ for $\alpha>1$ and $x,y>0$.
Question
Does it follow that whenever $f$ has increasing marginal product of input $x$, it also has increasing returns to scale?
Interesting question. I will provide two cases where the relation holds (and in the second one it is not obvious that it does), but I will refrain from saying that it holds for any conceivable production function. I will maybe return for a more general treatment.
CASE A
Consider the generalized Cobb-Douglas function in $n$ inputs
$$Q = A\prod_{i=1}^nx_i^{a_i},\;\;\ a_i>0$$
The "returns to scale" for this production function (formally, its degree of homogeneity), is $$r=\sum_{i=1}^na_i$$
If just one of the $a_i$'s is greater than unity, say $a_k$, the input $k$ will have an increasing marginal product (i.e. 2nd partial derivative positive,), and also, this will lead to $r>1$:
$$\frac {\partial^2 Q}{\partial x_k^2} = \frac {a_k(a_k-1)}{x_k^2}Q >0 \qquad r=a_k+ \sum_{i\neq k}^na_i >1 $$ So in this case, the implication holds: the existence of an input with increasing marginal product, implies that the production function will exhibit increasing returns to scale.
CASE B
Consider now a generalized C.E.S function, say, for two inputs,
$$Q= A\left[\alpha x_1^{-\rho}+(1-\alpha)x_2^{-\rho}\right]^{-k/\rho},\;\; 0<\alpha<1,\;\; \rho >-1$$
Where is the "generalization"? In the existence of the coefficient $k$, which determines the degree of homogeneity, and hence the returns of scale for this function ($k>1 \Rightarrow$ increasing returns to scale). The first derivative here is
$$\frac {\partial Q}{\partial x_1} = \frac {k\alpha Q^{(k+\rho)/k}}{A^{\rho/k}x_1^{\rho+1}}$$ and the 2nd derivative is
$$\frac {\partial^2 Q}{\partial x_1^2} = \frac {k\alpha}{A^{\rho/k}}\left[((k+\rho)/k)Q^{(k+\rho)/k}Q^{-1}\frac {\partial Q}{\partial x_1}x_1^{\rho+1} - (\rho+1)x_1^{\rho}Q^{(k+\rho)/k}\right]\cdot x_1^{-2(\rho+1)} $$
$$=\frac {\partial^2 Q}{\partial x_1^2} = \frac {k\alpha}{A^{\rho/k}}Q^{(k+\rho)/k}x_1^{\rho}\left[((k+\rho)/k)Q^{-1}\frac {\partial Q}{\partial x_1}x_1 - (\rho+1)\right]\cdot x_1^{-2(\rho+1)}$$
Setting $\varepsilon_{q,1} = Q^{-1}\frac {\partial Q}{\partial x_1}x_1$ (the point elasticity of production with respect to $x_1$, which for the CES production function is not constant) and ignoring the terms that do not affect the sign of the 2nd derivative, we have that
$$\operatorname{sign}\left\{\frac {\partial^2 Q}{\partial x_1^2}\right\}= \operatorname{sign}\left\{(1+\rho/k)\varepsilon_{q,1} - (\rho+1)\right\}$$
So for increasing marginal product, we need
$$\varepsilon_{q,1} > \frac {1+\rho}{1+\rho/k}$$
It would appear that this inequality can hold (=> increasing marginal product) even if $k\leq 1$ (in which case, the returns to scale will be constant or decreasing). But this is not so. We have
$$\varepsilon_{q,1} = Q^{-1}\frac {\partial Q}{\partial x_1}x_1 = Q^{-1}\frac {k\alpha Q^{(k+\rho)/k}}{A^{\rho/k}x_1^{\rho+1}}x_1 = \frac {k\alpha}{x_1^{\rho}}\left[\frac QA\right]^{\rho/k}= \frac {k\alpha}{x_1^{\rho}}\frac {1}{\left[\alpha x_1^{-\rho}+(1-\alpha)x_2^{-\rho}\right]}$$
$$\Rightarrow \varepsilon_{q,1} = \frac {k\alpha}{\alpha +(1-\alpha)x_1^{\rho}x_2^{-\rho}}$$
So the condition for increasing marginal product becomes
$$\frac {k\alpha}{\alpha +(1-\alpha)x_1^{\rho}x_2^{-\rho}} > \frac {1+\rho}{1+\rho/k} \Rightarrow \alpha k+\alpha\rho > (1+\rho)\alpha + (1+\rho)(1-\alpha)x_1^{\rho}x_2^{-\rho}$$
$$\Rightarrow 0 > (1-k)\alpha + (1+\rho)(1-\alpha)x_1^{\rho}x_2^{-\rho} $$
For this inequality to hold (and hence obtain increasing marginal product) it is necessary that $k>1$, (because $\alpha <1$ and $\rho >-1$), which implies increasing returns to scale. So if marginal product is increasing, then it is certain that we have increasing returns to scale, here too.