Consider a function $f$ defined over an interval $[a,b]$.
We say that $f$ is convex over $[a,b]$ if, for every pair of points $d,p$ in $[a,b]$ such that $d<p$, the slope of $f$ at $p$ is greater than or equal to the slope of $f$ at $d$.
So, as we sweep across $[a,b]$ from left to right, the slope of $f$ never decreases (and in the case of strict convexity, the slope is always increasing as we sweep across the interval).
And we say that $f$ is concave over $[a,b]$ if the opposite is true, i.e, if, as we sweep across $[a,b]$ from left to right, the slope of $f$ never increases (and in the case of strict concavity always decreases).
Is my understanding correct?
Yes, you are correct and your conclusions about the slope essentially give you an intuition for relating this to second derivative, as it is the derivative of a slope.