Distinct convex quadrilaterals formed by 12 points out of which 5 are collinear

Question

There are 12 points in a plane of which 5 are collinear. The number of distinct convex quadrilaterals which can be formed with vertices at these points is:___

I know how to solve this question without the convex part. How do I ensure that the quadrilaterals formed are convex? Or how do I count the number of concave quadrilaterals possible? Won't they depend on the real arrangement of points?

Note: It can be assumed that none other than these 5 points are collinear

Answer 1

I'll solve the problem under the following simplifying assumptions:

$S = A \cup B$ is a $12$-point subset of $\mathbb{R}^2$ such that

• $A = \{P_1,...,P_5\}$.
• $B = \{Q_1,...,Q_7\}$.
• The points of $A$ are collinear.
• For $k$ from $2$ to $4$, the point $P_k$ lies strictly between $P_{k-1}$ and $P_{k+1}$.
• No $3$ distinct points of $S$ are collinear unless they are points of $A$.
• The points $P_1,P_5,Q_1,...,Q_7$ are the successive counterclockwise vertices of a convex $9$-gon.

Call $4$ points of $S$ qualifying if they can be arranged to form the vertices of a convex quadrilateral.

Then with the stated assumptions, the count can be organized as follows . . .

• Any $4$ points of $B$ are qualifying, which gives a count of $\displaystyle{\binom{7}{4}}=35$.
• Any $4$ points, one from $A$ and three from $B$ are qualifying, which gives a count of $\displaystyle{5\binom{7}{3}}=(5)(35)=175$.
• Any $4$ points, two from $A$ and two from $B$ are qualifying, which gives a count of $\displaystyle{\binom{5}{2}\binom{7}{2}}=(10)(21)=210$.

Thus, the total count is $35+175+210=\boxed{420}$.