Consider the following two series:
$$a^{(0)}_n,\;\; n = 1,2,...,N \;\;\text{and}\;\; c_m,\;\; m=1,2,...,M$$
where both satisfy:
$$\sum \limits_{n=1}^{N}a^{(0)}_n = 1 \;\; \text{and}\;\; \sum \limits_{m=1}^{M}c_n = 1.$$
Then we can define $a_{nm}^{(1)} = a^{(0)}_nc_m$.
If we so choose, we can re-index this sequence by a bijection from $\{1, \dots , N \} \times \{ 1, \dots, M\} \rightarrow \{1, \dots, NM \}$, where $NM= N^{(1)}$. It is easy to see that: $$\sum \limits_{n'=1}^{N^{(1)}}a^{(1)}_{n'} = 1.$$ We can recursively define $a^{(j+1)}_{n'} = c_ma^{(j)}_{n}$ this way, and re-label again with a similar bijection from $\{1, \dots , N^{(j)} \} \times \{ 1, \dots, M\} \rightarrow \{1, \dots, N^{(j)}M \}$. This still satisfies:
$$\sum \limits_{n'=1}^{N^{(j)}}a^{(j)}_{n'} = 1 \;\;\forall j.$$
Question:
Is the following sum bounded as $j \rightarrow \infty$ when, for example, $a^{(0)}_n = \{0.6, 0.4, 0.4\}$ and $ c_m = \{0.46,0.27,0.20,0.07\}$?
$$\sum \limits_{n'=1}^{N'}a^{(j)}_{n'}\log(a^{(j)}_{n'}) $$
My Work
So I computed the profile of $a^{(j)}_n$ about as much as my RAM would allow...
This looks to me like the tail has a $\frac{1}{n^r}$ distribution, but I've been unable to prove anything. Any help is greatly appreciated!
Some Obvious Results
If $a_n = \frac{1}{N}$ and $c_m = \frac{1}{M}$ it is clear that the sum diverges, since it is equal to $\log(N^{(j)})$ (The entropy of a uniform distribution), which grows to infinity as $j \rightarrow \infty$. The real regieme where this question becomes interesting is when the $a_n$ and $c_n$ are closer to "geometric". It is known that the geometric distribution has finite entropy, and if $a^{(j)}_n$ can be bounded by some geometric sequence i.e $a^{(j)}_n < \frac{1}{n^r}$ as $j$ gets large for some $r > 1$ then we can conclude that the sum is convergent.
As commented by stochasticboy123,
You can view this sum as the entropy of a joint distribution of $A \sim a^{(0)}$ and j copies of $C \sim c$. Thus, the sum diverges linearly in $H(C)$.