I can't seem to wrap my mind around the concept of normalization. I am hoping these examples will clarify my understanding.
- If I have a variable A (which has 3 values eg something like A = Sunny, Rainy, Snowy) and 6 variables B, C, D and E, F, G (each have n values lets say) that depend on A. Then would the normalization factor be 1/ all values of A?
- Does the normalization factor always correspond to 1 over sum of all the values of the variable before the bar(in this case X) for P(X|Y, Z, W) If yes, why? If not, how do we decide the normalization factor?
By the definition of conditional probability: For any discretely distributed random variables $X,Y,Z,W$, we have (so long as $\mathsf P(Y{=}y,Z{=}z,W{=}w)>0$) that:$$\mathsf P(X{=}x\mid Y{=}y,Z{=}z,W{=}w)=\dfrac{1}{\mathsf P(Y{=}y,Z{=}z,W{=}w)}\cdot \mathsf P(X{=}x,Y{=}y,Z{=}z,W{=}w)$$
This fraction is called a normalising factor, because it ensures that the conditional probability mass function will have the normal behaviour of a probability mass function: ie We require the sum of probability mass functions over all possible states of $X$ to equal one.
$$\begin{align}1&=\sum_{x\in X(\Omega)}\mathsf P(X{=}x\mid Y{=}y,Z{=}z,W{=}w)\\[2ex]&=\dfrac{1}{\mathsf P(Y{=}y,Z{=}z,W{=}w)}\cdot \sum\limits_{x\in X(\Omega)}\mathsf P(X{=}x,Y{=}y,Z{=}z,W{=}w)\end{align}$$
Of course, this works because the Law of Total Probability states:
$$\mathsf P(Y{=}y,Z{=}z,W{=}w)=\sum\limits_{x\in X(\Omega)}\mathsf P(X{=}x,Y{=}y,Z{=}z,W{=}w)$$
(Really, they are all just different ways of expressing the same thing.)