Taking into account the Shannon entropy, I was wondering that, if we have a String like $1122344444455$ , is this possible to find out the entropy of digit $4$ in this String? In other words, I would like to know if we can find a way to measure the degree of uncertainty of occurrence of digit $4$ in this String. Is this the responsibility of Relative entropy?
2026-03-29 20:29:44.1774816184
Entropy of a character in a String
3.7k Views Asked by Bumbble Comm https://math.techqa.club/user/bumbble-comm/detail At
1
There are 1 best solutions below
Related Questions in INFORMATION-THEORY
- KL divergence between two multivariate Bernoulli distribution
- convexity of mutual information-like function
- Maximizing a mutual information w.r.t. (i.i.d.) variation of the channel.
- Probability of a block error of the (N, K) Hamming code used for a binary symmetric channel.
- Kac Lemma for Ergodic Stationary Process
- Encryption with $|K| = |P| = |C| = 1$ is perfectly secure?
- How to maximise the difference between entropy and expected length of an Huffman code?
- Number of codes with max codeword length over an alphabet
- Aggregating information and bayesian information
- Compactness of the Gaussian random variable distribution as a statistical manifold?
Related Questions in ENTROPY
- Relation between Shanon entropy via relation of probabilities
- How to maximise the difference between entropy and expected length of an Huffman code?
- Appoximation of Multiplicity
- Two questions about limits (in an exercise about the axiomatic definition of entropy)
- Computing entropy from joint probability table
- Joint differential entropy of sum of random variables: $h(X,X+Y)=h(X,Y)$?
- What is the least prime which has 32 1-bits?
- Eggs, buildings and entropy
- Markov chains, entropy and mutual information
- Entropy and Maximum Mutual Information
Trending Questions
- Induction on the number of equations
- How to convince a math teacher of this simple and obvious fact?
- Find $E[XY|Y+Z=1 ]$
- Refuting the Anti-Cantor Cranks
- What are imaginary numbers?
- Determine the adjoint of $\tilde Q(x)$ for $\tilde Q(x)u:=(Qu)(x)$ where $Q:U→L^2(Ω,ℝ^d$ is a Hilbert-Schmidt operator and $U$ is a Hilbert space
- Why does this innovative method of subtraction from a third grader always work?
- How do we know that the number $1$ is not equal to the number $-1$?
- What are the Implications of having VΩ as a model for a theory?
- Defining a Galois Field based on primitive element versus polynomial?
- Can't find the relationship between two columns of numbers. Please Help
- Is computer science a branch of mathematics?
- Is there a bijection of $\mathbb{R}^n$ with itself such that the forward map is connected but the inverse is not?
- Identification of a quadrilateral as a trapezoid, rectangle, or square
- Generator of inertia group in function field extension
Popular # Hahtags
second-order-logic
numerical-methods
puzzle
logic
probability
number-theory
winding-number
real-analysis
integration
calculus
complex-analysis
sequences-and-series
proof-writing
set-theory
functions
homotopy-theory
elementary-number-theory
ordinary-differential-equations
circles
derivatives
game-theory
definite-integrals
elementary-set-theory
limits
multivariable-calculus
geometry
algebraic-number-theory
proof-verification
partial-derivative
algebra-precalculus
Popular Questions
- What is the integral of 1/x?
- How many squares actually ARE in this picture? Is this a trick question with no right answer?
- Is a matrix multiplied with its transpose something special?
- What is the difference between independent and mutually exclusive events?
- Visually stunning math concepts which are easy to explain
- taylor series of $\ln(1+x)$?
- How to tell if a set of vectors spans a space?
- Calculus question taking derivative to find horizontal tangent line
- How to determine if a function is one-to-one?
- Determine if vectors are linearly independent
- What does it mean to have a determinant equal to zero?
- Is this Batman equation for real?
- How to find perpendicular vector to another vector?
- How to find mean and median from histogram
- How many sides does a circle have?
Entropy is a measure of how much information there is in one source with one probability distribution. Relative entropy is a measure how close or distant one probability distribution is to another. If you have one probability distribution you wish to test against your string then you could calculate a relative entropy.
Entropy can be defined as $$E(k) = \sum_{\forall i} -p_i\log_k(p_i)$$, which is the expected value of the logarithm of the probabilities. What is interesting here is that each $p_i$ occurs only in one term of this sum. So each outcome contributes with an own addition to the entropy.
It seems I need to clarify some words. If we consider (1,2,3,4,5) to be the "alphabet" ( in language of probability : the set of possible outcomes ). Each element (like "4") can be called a "symbol". Using these words instead of probability words are common to do in information theory. Which could be one of the tags the bot wanted us to add to the question.
The entropy of "4" is the expected number of bits on average we would need to store "4". 6 out of 13 symbols are "4". Let us for simplicity guess that 6 out of 12 or 7 out of 14 are "4". Then 4 would occur almost half of the symbols. The entropy (in binary digits) would then be $-\frac{1}{2} \log_2\left(\frac{1}{2}\right) = \frac{1}{2}$ which means that on average we would need 1/2 bits per symbol to consider if it was a "4" or not.
Having an entropy of 1/2 means that if "4" occurs on average 1 in 2 symbols we should need on average 1/2 bit per symbol to tell if it was a 4.
The thing with Relative Entropy or Kullback-Leibler Divergence is that it requires two distributions, we got to have some second distribution to compare against. Once we have that it's just to follow the formula on the wikipedia page. It's very straight forward.