Sometimes when I talk to my maths professors, they would say "this piece of maths (e.g. differential geometry) is hard". What do they exactly mean by "hard" (difficult) - when clearly they're the experts in the field and they've been doing it for years, even decades?
The thing is, I don't particularly find understanding maths books hard, although I do omit some exercises now and then while I revise. I also know the value of "hard work". The only thing I find hard in maths is when I pick up a maths book, but it assumes that I understand the terms used in it - when I have never seen them before.
There arre two definitions of "hard" in play here. Serious mathematicians label a problem "hard" if it is likely that a typical professional mathematician expert in the appropriate field could spend a career attacking that problem, perhaps developing a lot of new math, and still not solve it. For example, sphere-packing in more than 8 dimensions is hard; the Poioncare conjecture (now proven) was hard, almost any problem in Ramsey theory is hard, and there are loads of less famous hard problems which will probably eventually be solved (or proven undecidable within the usual axiom system). Erdos was famous for his ability to judge which problems were "too hard" and which were meaty but tractable.
Your professors are using "hard" in a different sense. What they are implying is that a your current level of mathematical maturity, you not only won't be able to master the subject, but strugglling to try would be an inefficient use of your efforts.
For example, I would not say differential geometry is inherently hard (being a theoretical physicist with a backgrund in General Relativity), but you certainly would not be doing anything useful in studying it before mastering vector calculus.