Let $f : [0, 1] → R$ be given by the formula
$f(x) = \frac{1}b$ if $x = \frac{a}b$, where $a$ and $b$ have no common factor,
$f(x) = 0$ if $x$ irrational.
Show that $f$ is continuous at $x$ if and only if $x$ is irrational. (Hint: Use the density properties of rationals and irrationals.)
My attempt: Let $\epsilon > 0$. Suppose $x$ is irrational, then $f(x) = 0$.
$|x-x| < \delta$
$|f(x) - x| = |-x| = x$
So we can choose $\delta = \epsilon$ so that $\epsilon > x$ (I'm unsure of this part).
Suppose $x$ is rational, $f(x) = \frac{1}b$. We want to show that is discontinuous. Suppose there is a $\delta$ that works for $|\frac{1}b-x| < \epsilon$. Let $\epsilon = \frac{1}2$. Let $x= \min(\delta, \frac{3}4)$
$|\frac{1}4 - \frac{3}4| = |-\frac{1}2| = \frac{1}2$
And $1/2$ is not greater than a $1/2$. So it is a contradiction.
Firstly we need to restrict the domain to $(0,1]$ or the function is not well-defined.
For any $x$ we can say $f$ is continuous at $x$ if for all sequences $(x_n)_{n=1}^\infty$ where $x_n \rightarrow x$ as $n \rightarrow \infty$ then $f(x_n) \rightarrow f(x)$ as $n \rightarrow \infty$. This defintion should be easier to use.
Suppose $x$ is irrational and $x_n \rightarrow x$ as described. If only finitely many $x_n$ are rational we note that $f(x_m)=0$ for high enough $m$ so we may assume without loss of generality that each $x_n$ is rational and $x_n=\frac{a_n}{b_n}$ where $a_n$ cannot divide $b_n$. If $b_n$ is bounded then this will force $b_m =b$ and $a_m =a$ for high enough $m$ (why is this true?) and $x=\frac{a}{b}$, thus $(b_n)_{n=1}^\infty$ is unbounded. If $(b_n)_{n=1}^\infty$ has a bounded subsequence then $(x_n)$ would have a limit point that is rational by a similar reason given above (yet again, why must this hold?) and so $b_n \rightarrow \infty$. It now follows that $f(x_n) = b_n \rightarrow 0 = f(x)$ as required.
Now suppose $x$ is rational. We now note that are the irrationals are dense in $\mathbb{R}$ there exists a sequence of irrationals $(x_n)$ where $x_n \rightarrow x$, then $f(x_n)=0$ and $f(x_n)$ doesn't converge to $f(x) \neq 0$, thus $f$ is not continuous at $x$.