What is the best way to average the frequency of Chinese characters in a sentence?

Question

I'm trying to find the best way to assign a "frequency score" to sentences in Chinese. Basically, I have a database which tells me how frequent each Chinese character is. From that, I would like to evaluate how "easy" a sentence is compared to other sentences. i.e. how easy it might be for a beginner to understand the sentence.

My first approach was to average all the character frequencies in the sentence. However, I found that certain sentences with very uncommon characters end up having a higher frequency score than sentences with more common characters. I think this is because it only takes a few very common characters to really increase the total score.

For example, here are two sentences of six characters (each number represents the frequency score of a character):

10 1 2 0 9 1 = 23

3 2 4 3 3 5 = 20

In this example, the second sentence is likely to be easier because all the characters are reasonably frequent. The first sentence has a higher score because of the two "10" and "9" characters. However, the "0" and "1" characters will make it hard to understand for beginners.

So I was wondering - what would be the best way to calculate the frequency score in this case?

Answer 1

Honestly, this has more to do with Chinese Language than Math.

Anyway, I would recommend taking the maximum percentile of the character for determining the difficulty of the sentence. As a learner myself, I would base my opinion of the sentence squarely on the most difficult portion of the sentence.

But this is just one opinion, you should probably play with different styles (range, mean etc)

Here are a few references which may help:

The database that you talk about might probably be a derivative of this or this.

In fact, the second link states:

Frequency-weighted average number of strokes:

• For the most frequently used 2,965 characters: 9.10;
• For the most frequently used 1,253 characters: 8.91;
• For the most frequently used 733 characters: 8.65.

The first link states:

100 characters → 42% understanding  1600 characters → 95.0% understanding
200 characters → 55% understanding  1700 characters → 95.5% understanding
300 characters → 64% understanding  1800 characters → 96.0% understanding
400 characters → 70% understanding  1900 characters → 96.5% understanding
500 characters → 75% understanding  2000 characters → 97.0% understanding
600 characters → 79% understanding  2100 characters → 97.4% understanding
700 characters → 82% understanding  2200 characters → 97.7% understanding
800 characters → 85% understanding  2300 characters → 98.0% understanding
900 characters → 87% understanding  2400 characters → 98.3% understanding
1000 characters → 89% understanding 2500 characters → 98.5% understanding
1100 characters → 90% understanding 2600 characters → 98.7% understanding
1200 characters → 91% understanding 2700 characters → 98.9% understanding
1300 characters → 92% understanding 2800 characters → 99.0% understanding
1400 characters → 93% understanding 2900 characters → 99.1% understanding
1500 characters → 94% understanding 3000 characters → 99.2% understanding

You could probably use this for your decision?

Also, have you considered just deciding on the basis of HSK Levels?