I need to compare 2 vectors i.e. assign probability on the basis of how similar they are. I tried using cosine similarity but it doesn't seem to give reasonable results. See below examples
Example 1: [10] [1]
Example 2: [4, 4, 4, 4, 4, 4] [0, 0, 1, 0, 0, 0]
Example 3: [6, 6, 6, 6, 6, 6] [0, 0, 1, 0, 0, 0]
Example 4: [90, 90, 90, 90, 90, 90] [0, 0, 1, 0, 0, 0]
Example 5: [90, 9, 90, 9, 90, 9] [0, 0, 1, 0, 0, 0]
Similarity :
1
0.40824829
0.40824829
0.40824829
0.57448499
As we can see vectors in Example-2,3,4 are not similar but their cosine similarity is same and also vectors [90, 90, 90, 90, 90, 90] [0, 0, 1, 0, 0, 0] are not very close as vector 2 has only 1 non zero value but still similarity between then is assigned to around 40%.
Here's a suggestion, use the following similarity measure
$$S(x,y)=1-\frac{\|x-y\|}{\|x\|+\|y\|}$$
in which
$$\|x\|=\sqrt{x_1^2+x_2^2+\ldots+x_n^2} \; .$$
This gives for your examples
By the triangly inequality, we have that
$$\|x-y\|\leq\|x\|+\|y\|$$
and thus $S(x,y)\geq 0$ and since lengths are positive we also have that $S(x,y)\leq 1$.
When $x=y$, $S(x,y)=1$ and when $x=-y$, $S(x,y)=0$.