I am looking at web page to tell what is its content (the main text part) and its title.
I can estimate if I am looking at a title or content by the page's semantics, but wanted to add a rules that look at the lengths. So I am looking for something like the graph on picture below.
The graph should approach y = 1 at x = average_length and then slowly die away.
Ideally, I would like to specify the average content length from min to max. Maybe a spline?
Would love to hear if you had similar task at hand and how you treated the text length.
Thanks!
You probably want a Gamma distribution, or an Erlang distribution.
They have the kind of shape you showed, and the Gamma distribution has been used to describe lengths of words and sentences, so it seems suitable. Details here.
If these probability density functions are too complex for your application, simple approximations could probably be constructed. Ask again if that's what you want to do.