Maximize the area of a convex shape in a right isosceles triangle under some restrictions

Question

For a given point (P) in an right isosceles triangle (see Figure), what is the curve or polygon that complies with the following:

i) One of its endpoints is at A

ii) Its other endpoint is at B

iii) It crosses through point P

iv) It cannot be outside the boundaries of triangle ABC (it can lie on its boundaries though)

v) It creates a convex shape with AB.

AND also complies with one of the following:

1. It encompasses the maximum possible area

OR

2. It encompasses the minimum possible area.

I guess that for (2) one simply needs the convex hull of the points APB. Not sure how things look like for the (1) though.

Answer 1

Hint : (too long for a comment).

Have a look at the figure below (which is a rotated version of your figure with natural coordinates) :

Consider a mobile line passing through $P(x_0,y_0)$ with equation $y=y_0-m(x-x_0)$ (with $m>0$). It will delimitate with the coordinate axes a triangle $ODC$.

Consider the case where $0 \le x_0 \le \tfrac12, 0 \le y_0 \le \tfrac12$.

In this case, and in this case only, it is not difficult to prove that the area of triangle $ODC$ is minimal when $m=\tfrac{y_0}{x_0}$. Therefore, (convex) quadrilateral $DBAC$ will have a maximal area (equal to $\tfrac12-2x_0y_0$). In this case, the solution curve is broken line $DBCA$.

Can you consider the other cases on this basis ?