Here in the U.S., it is my experience that over 75% of adults I meet socially will volunteer that phrase or a variation upon learning that I am a mathematician. I find this frustrating, since almost nobody would brag about being bad at history or English or really any other subject. Normally I change the subject, or perhaps say they must have had a bad classroom experience, but that feels unsatisfying and turns the conversation in an ugly direction.
Surely some of the bright minds here on Math.SE have found a good response. Note that I'm not looking to alienate a new acquaintance, so let's keep it positive.
Math is hard for everyone. It takes long hours of practice to become competent. I think that's what people want to hear; and fortunately, it's true.
This reply has the advantage of steering the conversation in a somewhat more positive direction---in my experience, usually towards whatever it is that the person does like to do and has practiced long hours at. This is usually more interesting than having a conversation about poorly taught math courses.