Problem: Show that $I(A)$ is not compact in $(X, \delta)$.
Let $X$ be a set of convex polygons in a plane, and for all $A \in X$, $m(A)$ denotes the area of $A$.
We define a distance function on $X$ by
$$\delta(A,B) = m((A \cup B) - (A \cap B))$$ for $A, B \in X$. If we use the notation $\triangle$ as the symmetric difference of $A$ and $B$, we can write $\delta(A, B) = m(A \triangle B)$.
I checked $\delta$ is exactly a distance function on $X$. Therefore $(X, \delta)$ be a metric space.
For $A \in X$, define $I(A) = \{ E \in X \hspace{1mm}| \hspace{1mm} E \subset A\}$. I want prove $I(A)$ is not compact in $(X, \delta)$.
Actually I also proved that $m : X \rightarrow \mathbb{R}$ is continuous on $X$, thus I tried 2 ways to prove this problem but both didn't work:
If $I(A)$ is compact in $X$, $m(I(A))$ should be also compact in $\mathbb{R}$, but it is obviously $m(I(A)) = [0, m(A)]$ because we can choose an(or more) appropriate subset $E$ of $A$, which satisfies $m(E) = y$, for any $y \in [0, m(A)]$. Thus I couldn't get anything from this approach.
If $I(A)$ is compact in $X$, because $X$ is a hausdorff space, $I(A)$ is closed in $X$. Thus I tried to lead a contradiction from here(e.g., $I(A)$ is not closed in $X$), but I couldn't. Because $X$ is a metric space, I can write the definition of open sets in $X$ precisely: $$U \subset X \hspace{1mm} \text{is open} \Leftrightarrow \hspace{1mm} \text{For any} \hspace{1mm} u \in U,\hspace{1mm} \text{there exists} \hspace{1mm} \varepsilon >0 \hspace{1mm} \text{satisfies} \hspace{1mm} B(x, \varepsilon) \subset U$$ Where $B$ is a open ball in $X$. But it looks there is no contradiction in this approach, too.
I don't know what I should do next. How can I show the first statement?
I'll try at an answer, given what you commented so far.
Since $X$ does not contain points, I would say that you can find a Cauchy sequence of convex polygons whose limit is not a convex polygon.
Take a fixed convex polygon $A_0\in X$ with $m(A_0)>0$. Find a sequence of convex polygons, $\{ B_n\}\in I(A_0) $ , such that $\text{diam}(B_n)\to 0$. You can find a subsequence of nested convex polygons $\{ B_{n_k} \}$, since $A_0$ is compact.
By Cantor's intersection theorem, there exists a $p\in \text{int}(A_0)$ such that $\cap_{k}B_{n_k}=\{p \}$. For $k_2> k_1$, we have that $\delta( B_{n_{k_1}},B_{n_{k_2}} )= m( B_{n_{k_1}} )\leq \pi \big(\text{diam}(B_{n_{k_1}}) \big)^2 \overset{k_1\to \infty}{\to}0$.
Since, by what you said, any convex polygon will have non-negative area it follows that there is no limit to the subsequence.
I've skipped some details because I thought they would be tedious. I think you also need to clarify some details about $X$. Namely, is $X$ the set of all convex polygons, and is $I(A_0)$ infinite. I think that this probably the direction to show non-compactness.