I want to show that there is not a continuous function $f:D\to R$ such that
$f(\alpha u+\beta v)=\alpha f(u)+\beta f(v)$, with $\alpha+\beta=1$, $\alpha,\beta\geq 0$.
$f(0,1)=0$.
- $f(u)>0$ whenever $u\neq (0,1)$.
and $D$ is the convex hull of { $(x,y)\in R^2: x^2+y^2\leq 1$}$\cup${$(1,1)$}.
I think that if there is that function it should have the following property: $f(1,1)=0$ but I cannot prove this. I would like to obtain any suggestion. Thanks.
Let's exploit the structure of $f$ and argue by contradiction. Suppose all three properties are met. Let $c=f(0,0)$ and $d=f(1,0)$. If any of $c$ or $d$ are non-positive, the third condition is violated and we're done, so we will assume $c>0$ and $d>0$. The convexity property of your function immediately determines the values of $f$ in the triangle $T$ with vertices $(0,0), (1,0), (0,1)$.
However, there is more! Now, pick any point $p$ in $D$ which isn't in $T$ (for instance, the corner $(1,1)$). To find the value of $f(p)$, we pick an auxiliary point $q$ in the interior of $T$. Because the interior of $T$ is an open set, we are able to pick another point $r$ which lies in the segment $p$-$q$ but still lies in the interior of $T$; this means we know $f(q)$ and $f(r)$, and we can write $r = \alpha p + \beta q$, and the convexity condition of your function gives us a linear equation for the value of $f(p)$. In this manner, the values of $f$ everywhere in $D$ are determined uniquely by the choice of $c$ and $d$.
Now that we know $f$ everywhere, write \begin{align} f(x, y) &= f(x, y\times 1 + (1-y) \times 0) \\&= yf(x, 1) + (1-y)f(x,0), \\&= y[f(x\times 1+(1-x)\times 0, 1)] + (1-y)[f(x\times 1+(1-x)\times 0, 0)], \\&= y[xf(1,1) + (1-x)f(0,1)] + (1-y)[xf(1,0) + (1-x)f(0,0)], \\&= xyf(1,1) + (1-x)yf(0,1) + x(1-y)f(1,0) + (1-x)(1-y)f(0,0), \\&= xyf(1,1) + x(1-y)d + (1-x)(1-y)c, \end{align} for any $(x,y)\in D$. We don't know $f(1,1)$ explicitly, but we can use convexity to find its value. $(0.5,0.5)$ is the midpoint of $(1,0)$ and $(0,1)$, so $f(0.5,0.5)=d/2$. Applying the same argument again, $(f(1,1)+c)/2 = d/2$, which yields $f(1,1)=d-c$, and if $c\geq d$, we're done! If not, we write, at last: \begin{align} f(x, y) &= xy(d-c) + x(1-y)d + (1-x)(1-y)c, \\&= d[xy + x(1-y)] + c[(1-x)(1-y) -xy], \\&= dx + c(1-x-y). \end{align}
Just for a moment, let's extend $f$ to the whole plane, and let's find its zeroes: \begin{align} 0 &= dx + c(1-x-y), \end{align} which yields \begin{align} y=\frac{d-c}{c}x+1. \end{align} This line goes through the point (0,1), as expected. However, since we assumed $c<d$, the line has a positive slope, and thus intersects $D$ in the $x<0$ region, which gives the final contradiction.
I hope this helps!