So we have a piecewise function:
I've show it is continuous at x=0 and now I am trying to express in mathematical terms (if possible) that the derivative does not exist because:
The derivative is defined as the limit: $$f'(x)=\lim\limits_{\Delta x\to 0} \frac{f(x+\Delta x) - f(x)}{\Delta x}\tag{1}$$ Thus, for x = 0; $f'(0)=\frac{f(\Delta x)-f(0)}{\Delta x}$
Since, for any $\Delta x$ we choose so that $f'(0)$ is rational there exists a smaller $\Delta x$ such that $f'(0)$ is irrational, and vice versa, it follows that as $\Delta x$ tends to 0, $f'(0)=1$ or $f'(0)=0$. The limit/derivative is therefore not defined.
I feel like there is a better way to express this. I don't want to prove it just express it more mathematically.
Also, $ f(x)=\begin{cases} x, \, \text{ if } x \text{ is rational }\\ 0, \, \text{ if } x \text{ is irrational } \end{cases} $
Notice the following, $$f'(0)=\lim\limits_{\Delta x\to 0} \frac{f(\Delta x) }{\Delta x}$$
For rational $\Delta x$ we get $$\frac{f(\Delta x) }{\Delta x}=1$$
And for irrational $\Delta x$ we get $$\frac{f(\Delta x) }{\Delta x}=0$$
Thus the limit does not exist.
Intuitively you have two different slopes at $x=0$ which is not good.