I have been learning about statistics and probability theory superficially over last two-sih years, and on many occasions I have stumbled upon statements like: "The covariance gives some sense of how much two values are linearly related to each other, as well as the scale of these variables"
Now my question is basically in two parts. First part of my question is, what is actually meant when such a statement is made? What exactly is it that covariance is telling us about relationship between the variables? How does all of it relate to linearity?
The second part of my question is basically if this idea can be extended and generalized. Can we define some kind of higher covariance which captures non-linear aspects of relationship between the variables?
I know this question is quite vague, but the core here is to understand what such statements mean, and what exactly is it that they are telling us, both formally and intuitivelly.