Show that (1) $A(S) = \{Ax : x \in S \}$ and (2) $A^{-1}(S) = \{x : Ax \in S \}$ are convex but (1) is not always closed while (2) is closed.

Question

Problem definition:

Show that image of $S$ under $A$, i.e., $A(S) = \{Ax : x \in S \}$ is (a) convex and (b) but not closed in general if $S \subseteq \mathbb{R}^n$ is convex. Also, can we show when $A(S)$ is closed and when it is not?

However, if we consider the preimage $S$ under $A$, i.e., $A^{-1}(S) = \{x : Ax \in S \}$, then it is both closed and convex.

EDIT: $S \subseteq \mathbb{R}^n$ is convex and $A \in \mathbb{R}^{m \times n}$.

EDIT EDIT: Additionally, assume that $S$ is closed, i.e., $S$ is a closed and convex set.

Answer 1

(1) I suspect to your proof of convexity. You need to prove that$$\text{if }\quad v_1,v_2\in A(S)\quad\text{ then }\quad \alpha v_1+(1-\alpha )v_2\in A(S)\quad,\quad0\le \alpha \le1$$if $v_1,v_2\in A(S)$ then there exist $x_1,x_2\in S$ such that$$v_1=Ax_1\\v_2=Ax_2$$also $S$ is convex so $\alpha x_1+(1-\alpha )x_2\in S$ which means that $$A(\alpha x_1+(1-\alpha )x_2)\in A(S)$$and $A$ is linear therefore$$A(\alpha x_1+(1-\alpha )x_2)=\alpha Ax_1+(1-\alpha)Ax_2=\alpha v_1+(1-\alpha)v_2\in A(S)$$which completes the proof on convexity.
We prove the following theorem $$\text{A}(S)\text{ is closed if and only if }S\text{ is closed.}$$proof 1: let $S$ be closed. Then every convergent sequence $v_1,v_2,v_3,\cdots \in A(S)$ converges to some $v$ whether in $A(S)$ or not. By definition we have $$v_1=Ax_1\\v_2=Ax_2\\v_3=Ax_3\\.\\.\\.$$since $v_i$s converge and $A$ is linear so the sequence $x_1,x_2,x_3,\cdots$ is convergent to some $x$ and because S is closed hence $x\in S$ which is equivalent to $Ax\in A(S)$. Therefore the sequence $v_1,v_2,\cdots \in A(S)$ converges to $v=Ax\in A(S)$ and $A(S)$ is closed.
proof 2: let $S$ is not closed and assume by contradiction that $A(S)$ is closed. Therefore there exists some convergent sequence $x_1,x_2,x_3\cdots \in S$ converging to some point outside of $S$. This sequence can be translated to $v_1=Ax_1,v_2=Ax_2,v_3=Ax_3,\cdots \in A(S)$ which is convergent to $v\in A(S)$. Hence the definition $$\exists x\in S\quad,\quad v=Ax$$therefore the sequence $x_1,x_2,x_3\cdots \in S$ is convergent to $x\in S$ which is contradiction. Therefore $A(S)$ is not closed and the proof is complete.
(2) The proof of convexity is simple, but I'm not sure if $A^{-1}(S)$ is always closed. Here is a counterexample. Take $A=I$ and $S$ any convex open set (such as inside the unit ball). Then $A^{-1}(S)=\{x|x\in S\}=S$ which is convex but open.

Answer 2

(b) I assume that $S$ is a closed convex set; otherwise, an example is too trivial. I claim that, for a noninvertible nonzero linear transformation $A:\mathbb{R}^n\to\mathbb{R}^m$, there exists a closed convex set $S$ such that $A(S)$ is not closed. Indeed, we have the following proposition.

Proposition. Let $A:\mathbb{R}^n\to\mathbb{R}^m$ be a linear transformation. Then, for every closed set $S$, the set $A(S)$ is closed if and only if $A=0$ or $A$ is invertible. (The convexity of $S$ is unnecessary.)

If $A$ is noninvertible and nonzero, then there exists a nonzero vector $v\in\mathbb{R}^n$ such that $Av=0$. Since $A$ is nonzero, there exists a nonzero vector $u\in\mathbb{R}^n$ perpendicular to $v$ with respect to the usual Euclidean norm on $\mathbb{R}^n$. Define $$S:=\big\{xu+yv\,\big|\,x,y\in\mathbb{R}_{>0}\text{ and }xy\geq 1\big\}\,.$$ It can be easily seen that $S$ is a convex set. However, $A(S)=\big\{t\,Au\,|\,t\in\mathbb{R}_{>0}\big\}$ obviously is not a closed set (since $0$ is a limit point of $A(S)$ that does not lie in $A(S)$).

If $A$ is zero, then $A(S)=\emptyset$ or $A(S)=\{0\}$, which are closed sets. If $A$ is invertible, then $m=n$ and $A$ is a $\mathcal{C}^\infty$-diffeomorphism from $\mathbb{R}^n$ to $\mathbb{R}^m$, so $A$ sends closed sets to closed sets (and $A$ sends convex sets to convex sets due to Part (a)).

However, if you have a specific closed set $S$ and you want to determine whether $A(S)$ is also closed, then it is also possible to answer. First, let $N$ denote the nullspace of $A$. Then, $A:\mathbb{R}^n\to\mathbb{R}^m$ factors through the canonical projection $\pi:\mathbb{R}^n\to\mathbb{R}^n/N$ via $\alpha:\mathbb{R}^n/N\to\mathbb{R}^m$ sending $v+N\mapsto Av$ for any $v\in\mathbb{R}^n$ (that is, $A=\alpha\circ\pi$). Note that $\alpha$ is a $\mathcal{C}^\infty$-diffeomorpism from $\mathbb{R}^n/N$ to its image. Therefore, it suffices to determine whether $\pi(S)$ is a closed set.