Yes, I know, 20/100=20%, so there is 20% improvement if someone or something run the same distance or finish the same task from 100 seconds to 80 seconds.
However, recently I saw another argument says there should not be 20% but 40%, because from 100s to 50s, that's 100% improvement. 20s is 40% of 50s, so there is 40% improvement.
So where goes wrong?
Unfortunately, neither result you stated is correct. Note speed and time are different things, with them being inversely proportional to each other (i.e., as one increases by a factor of $f$, the other decreases by a factor of $f$). For example, let's say the average speed is how fast an object moves a distance $d$. Using that speed is the distance divided by the time, i.e., $s = \frac{d}{t}$, this then gives
$$s_1 = \frac{d}{100} \tag{1}\label{eq1A}$$
$$s_2 = \frac{d}{80} \tag{2}\label{eq2A}$$
as the speeds before and after. The relative change, with a negative value meaning a decrease, when expressed as a percentage at the end, is given by
$$\begin{equation}\begin{aligned} c & = \frac{s_2 - s_1}{s_1} \\ & = \frac{\frac{d}{80} - \frac{d}{100}}{\frac{d}{100}} \\ & = \frac{100}{80} - 1 \\ & = .25 \\ & = 25\% \end{aligned}\end{equation}\tag{3}\label{eq3A}$$
This is what Doug M's question comment also states.
The reason the other argument you saw also doesn't work is that it's assuming speed and time are directly proportional (i.e., as one increases by a factor of $f$, so does the other) but, as I stated earlier, they are actually inversely proportional instead.