How important is the "prestige" of your university for undergraduate mathematics?

Question

I go to a university in Canada. It's not really considered a top university (it is one of the best universities in the province but is ranked really low nationally). I study mathematics here and didn't bother applying to universities outside of the province because I wanted to stay close to my family. How important is the "prestige" of a university to your ACTUAL learning of mathematics (in terms of content, quality, knowledge, workload, etc)? I feel like I will be behind the pace of those at a top national university or those big American schools like M.I.T., Caltech, Harvard, etc as the years go by. Is there really a significant difference? I asked this question because I have several American friends that go to a state university, and one that goes to one of the top universities (in his case, M.I.T.) and we always joke about the level of difference between us. But one day, we got serious and we wanted to see if there was a big gap in our knowledge, and there wasn't. But I wanted to see others' feedback on this topic.