In an information retrieval system, I have a vector of values which represent occurrences of a term in documents:
Term is preferred when it appears more frequently in very few documents. So, I need a numerical value to prefer vector which has few high peaks. For example:
TERM1 is preferred over TERM5 because it has one high peak compared to 4 high peaks
TERM1 is preferred over TERM6 because it has one high peak compared to 1 low value
TERM4 is preferred over TERM5 because it has one high peak and 3 low values compared to 4 high peaks
etc...
Is there any formula or calculated value that can give higher weight to preferred vectors? p.s. I am not a mathematician
Thanks in advance
In statistics, there are different measures of concentration, which you could calculate for each term and then rank the terms based on the concentration measure. For example, you could use the Herfindahl index. It works as follows:
For each term $i=1,2,3,4,5,6$ and for each document $j=1,2,4,5$ (i.e., for each cell in your spreadsheet) you calculate the following:
$$s_{i,j}:=\frac{\text{frequency of term $i$ in document $j$}}{\text{sum of frequencies of term $i$ across all documents}}.$$
Then, you calculate for each term $i=1,2,3,4,5,6$ the Herfindahl index: $$H_i:=\sum_{j=1}^5 s_{i,j}^2=s_{i,1}^2+s_{i,2}^2+\ldots+s_{i,5}^2.$$
The index ranges from $\frac{1}{5}=0.2$ to 1. A higher number would be preferred in your example. Hence, you rank then the six terms based on the index in descending order starting with the highest index.
For example, term 1 has an index of $H_1=1$ and term 5 has an index of $$H_5=\frac{6^2}{27^2}+\frac{8^2}{27^2}+\frac{6^2}{27^2}+\frac{7^2}{27^2}+\frac{0}{27^2}=\frac{36+64+36+49}{27^2}=\frac{185}{729}\approx 0.25<H_1=1.$$ Hence, term 1 is preferred over term 5.
There is one problem: The index only measures concentration. So, both term 1 and term 6 have an index of 1. To resolve ties, you could rank ties based on the highest peak (e.g., term 1 is preferred over term 6).
Edit: If you prefer that the index ranges from 0 to 1 (e.g., to make it comparable to an index calculated for a different number of documents), you can normalise it via $$H^*_i:=\frac{H_i-1/n}{1-1/n},$$ where $n>1$ is the number of documents (in your example $n=5$).