One of my colleagues argues that everything in math proves something in the physical world. For instance, he claims that the existence of math to describe fractals proves the infinite divisibility of matter (which is a non sequitur conclusion to make).
What is some simple mathematical concept that absolutely does not map to the physical world, contrary to this belief?
No one knows the true nature of the physical world. However, recent work seems to suggest that it is quite discrete. That is, things like real numbers may not represent anything real. Also, it seems likely that the entire observable universe can exist in only finitely many possible states. If this is so, then likely nothing infinite in mathematics represents reality.
If you don't want to go to such extremes, I think you will still find it extremely difficult to find anything in the real world relating to even fairly small ordinal numbers, like $\omega_1$, or even fairly small cardinal numbers like $\aleph_2$.