We assume that $X_i \sim N(μ_1, σ_1^2)$ and $Y_i \sim N(μ_2, σ_2^2)$ [i.e. that they are normal distributions with a finite (or known?) mean and variance] [...]
-- Tamhane & Dunlop, Statistics and Data Analysis, Section 8.3.2 Matched Pairs Design, p. 283.
Am I correct in understanding that because my two distributions are not normal (see image below), paired and unpaired t-tests cannot be performed on them, and therefore I should use the ranksum test instead to determine statistical significance?

Is it enough for their difference to be mostly normal, or must the distributions themselves also be normal? Looking at Tamhane & Dunlop's discussion, it appears to me their difference has normal distribution properties only if the distributions themselves are normal, and hence I cannot do a paired t-test with this data.
Am I correct? Please correct any misunderstandings I have. I would also appreciate additional free online resources to read.