Likelihood formulation

Question

I try to understand the relation of likelihood to cross-entropy by reading cross-entropy.

The problem is I cannot understand the formula for the likelihood in the article. The likelihood is defined as follows

$$\prod_{i}^{}q_i^{Np_i}$$

where

$q_i$ is the estimated probability of outcome $i$, $p_i$ is the empirical probability of outcome $i$ and $N$ is the size of the training set.

I haven't seen the formulation of the likelihood like that before that combines estimated and empirical probabilities. Why $p_i$ takes place in the formula? What's the motivation behind this formulation?

Answer 1

This is because in a block of length $N$, outcome $i$ appears about $Np_i$ times and the probability of each of them is estimated as $q_i$. This is why for each outcome it is equal to $q_i^{Np_i}$.

Answer 2

It just looks to me like the person who wrote the wiki article isn't phrasing themselves all that greatly.

Your model is that $(X_n)_{1\leq n\leq N}$ are iid and $\mathbb{P}(X_n=i)=q_i$. So given a realisation $(x_n)_{1\leq n\leq N}$, we have that the likelihood function is exactly

$$ \ell_x(q)=\mathbb{P}(X=x|q)=\prod_{n=1}^N q_{x_n}=\prod_{i} q_i^{\#\{n| X_n=i\}}, $$ and the exponent here is exactly the empirical probability distribution multiplied by $N$.