I'm having an argument with someone who thinks it's never justified to teach something that's not strictly correct. I disagree: often, the pedagogically most efficient way to make progress is to iteratively learn and unlearn along the way.
I'm looking for examples in mathematics (and possibly physics) where students are commonly taught something that's not strictly true, but works, at least in some restricted manner, and is a good way to understand a concept until one gets to a more advanced stage.

Thinking of Dirac delta function as a function works reasonably well up to a certain point. For example, every physicist knows that $$\int_{-\infty}^{\infty}e^{i\omega x}dx=2\pi\cdot\delta(\omega),$$ but only a small part of them really studied the theory of distributions.