For any $a,b \in S$ there exists $c \in S$ such that $a,b,c$ form an arithmetic progression in some order.
Prove that $| S |\leq5$.
I am struggling to find examples where it works.
I found a very simple example of $\{1,2,3\}$. I tried solving the contrapositive but that seemed to complicate things even further. How should one start when going about solving this?
Let $S$ be such a set with $|S|>1$. As the property of $S$ is invariant under translation and scaling, we may assume wlog. that $\min S=-1$, $\max S=1$. For every positive $x\in S$ we must have $\frac{x-1}2\in S$ in order to form a progression containing $-1$ and $x$. For every negative $x\in S$ we must have $\frac{x+1}2\in S$ in order to form a progression containing $1$ and $x$. We can summarize this as $$x\in S\implies f(x) \in S $$ where $$f(x)=\frac{x-\operatorname{sgn}(x)}{2}.$$ In particular, $0=f(\pm1)\in S$.
Note that $f$ restricted to $(0,1)$ is an injective map $(0,1)\to(-1,0)$; likewise it is an injective map $(-1,0)\to (0,1)$, hence an injective map from $(-1,0)\cup(0,1)$ to itself. But $f$ also maps $S\to S$. Hence if we let $T=S\setminus\{-1,0,1\}$ then $f$ is an injective map $T\to T$. We conclude that $f\colon T\to T$ is bijective, and so is $f\circ f$. In particular, for any $x_0\in T$, the sequence defined recursively by $x_{n+1}=f(f(x_n))$ must be periodic. If $x>0$ then $$f(f(x))=f(\tfrac{x-1}{2})=\frac{\tfrac{x-1}{2}+1}{2}=\frac{x+1}{4}$$ Thus for $x_0>0$ the sequence $(x_n)$ converges to the unique fixed point of the map $x\mapsto\frac{x+1}{4}$, i.e., $x_n\to \frac13$. A convergent periodic sequence must be constant, whence $x_0=\frac13$. Similarly, for $x<0$ we have $f(f(x))=\frac{x-1}4$ and conclude that the only possible negative element of $T$ is $-\frac13$. It follows that $T\subseteq \{-\frac13,\frac1\}$ and so $S\subseteq \left\{-1,-\frac13,0,\frac13,1\right\} $ or (undoing the normalization from the beginning) $$S\subseteq \left\{a-3d,a-d,a,a+d,a+3d\right\} $$ with $a\in\Bbb R$ and $d>0$.
Remark. Revisiting the argument above and noting that $f(\pm\frac13)=\mp\frac13$, there are only the following possibilities (in particular, $|S|$ is neither $2$ nor $4$): $$\begin{align}S&=\emptyset\\ S&=\{a\}\\ S&=\{a-d,a,a+d\}\\ S&=\{a-3d,a-d,a,a+d,a+3d\}\end{align} $$ with $a\in\Bbb R$, $d>0$.