My text book says that the following function is not a valid PDF:
According to my calculation this could be a PDF if $k=3/2$
I applied integration to $f(x)$ between $1$ and $3$, and found $3/2$.
Why is this function not a valid PDF?
What am I missing here?


One way could be this. Fixing a k, you can define your PDF. So if you show that for every k, there exists an interval I in [1,3] where your integration is negative ($\int_{A}{kx(x-2)}dx = P_{k}(X \in A)<0$), that pdf is not a density because the probability of an event could not be negative. So if k<0 then I=[2,3], if k>0 then I=[1,2]. You have just calculated the k for which your integral is 1, that in general is not sufficient for defining a PDF.