I know for sure which numbers appear more often in a 5 out of 36 lottery (5 unique numbers out of pool of 36 numbers ranging [1,2,…,36]).
I know that 4 balls appear more often than the average, 11 balls less often. About other balls I do not have any reliable information - they are average. Max difference between often and rare balls is 110 hits (in 8000 trials), difference between rare and average and often and average is around 50 (in 8000 trials). Not so big, but it is consistent in any dataset (take any subset of any trials and it is always there unaltered).
What is the best stake with this knowlege?
If I have to stake on 5 numbers - I'd better stake on 4 numbers from "often" group" and 1 number from "normal" group, always avoiding to stake on rare numbers. On the other hand, the difference in frequency is not that big and "rare" numbers (they are 11 !!!) would be appearing around a lot, and they would surely appear often in combinations produced by lottery machine. So I feel that 100% avoiding rare numbers might be even a worse strategy.
Could you help me clear up my confusion and confirm the best strategy in this case?
P.S. So I counted the probability that 5 numbers (in one play) don't have any of 11 rare numbers.
One play (single draw with 5 numbers) is equivalent of 5 Bernoulli trials (tossing a single "dice" with 36 sides). Each next trial (appearance of next number) is a conditional probability, because trial 2 is equivalent of tossing a single "dice" already with 35 sides (36 - what appeared in trial 1). So in trial 1 probability to get a number within rare group is 11/36, probability to get a number not in rare group is 1-11/36. In trial 2: probability to get a number within rare group is 11/35 (UNLESS THAT NUMBER APPEARED IN TRIAL 1 ! Then it is 10/35), probability to get a number not in rare group is 1 - 11/36. Finally, probability that all 5 numbers (results of all 5 trials) are not from rare group is just a product of respective probabilities of trials 1-5: (1 - 11/36)(1 - 11/35)(1 - 11/34)(1 - 11/33)(1 - 11/32) = 14%.
On the other hand, probability that a ball from rare group would appear at least once in 5 trials is 86% (100% - 14%).
So not using rare balls is definitely a bad idea. What is optimal though?
This is not an answer but some general comments.
We could indeed consider this problem from a viewpoint that the lottery balls are biased and try to define an optimal strategy. Of course the strategy is simply to find out which of the $\binom{36}{5}$ combinations of possible draws is the most probable. This is however not as easy as it seems, in fact one would need more information on the system or make several assumptions that can not be validated at this moment.
The first thing we need to address is the question whether there is a reasonable argument to be made for a biased lottery game or not. Also in a fair lottery game weird streaks will occur, similar to the possibility that one can throw 10 heads in a row in a fair coin toss. Its probability is only 1/1024, but if we play it often enough it will occur. So do we have enough reason to suspect that the game is biased?
The data you present at this moment is too limited to do a proper analysis. It is also a bit unclear whether you have 8000 trials of drawing a single ball from 36, or that these are 8000 actual games in which 5 balls are drawn in succession. With those details and the actual numbers of each ball occurring, one can calculate the confidence for the game to be fair or not.
For instance if you actually have the data of 8000 draws of 5 balls, we can do a fast check. Working from the hypothesis that the game is fair, a particular ball would have a chance 5/36 to appear within a draw. With 8000 games this would mean that on average a ball should appear 1111.11 times. So what is the probability $P(n \leq 1060)$ that ball no. 1 appears less then 50 times of what is expected? $$P(n \leq 1060) =\sum_{n=0}^{1060} \binom{8000}{n} \left(\frac{5}{36}\right)^n \left(\frac{31}{36}\right)^{8000-n} \approx 0.0502$$ With 36 balls, the probability that such a ball would not exist is $(1 - 0.0502)^{36} \approx 0.1566$, and hence finding one or several of such balls is to be expected.
A more careful analysis could be made from the full data set as well as knowledge from how the data set was obtained.
If we assume that the game is biased, we are faced with the problem of how it is biased and it is going to affect the analysis as well as the formulation of a strategy to exploit such a bias. There are numerous methods to bias such a game, for instance making the balls unequal in size/weight/shape or other properties, but also an improper initial mixing. For instance in an inexperienced/poor shuffle of an ordered new deck of cards makes certain cards more likely to be close together or to be found near top/bottom of the deck.
From a mathematical point of view we would need the probability of a ball to be drawn at each round of the 5 draws. In a fair game all balls are identical and there is no preference and those probabilities are trivial, but in a biased game this will be different. It could even be that the bias only occurs in the first draw, but not in the later rounds (insufficient mixing). Without such knowledge, one could not formulate a valid strategic choice, but with that information the most likely combination or combinations can be obtained and your chance of winning will be higher than a random choice. Actually winning is something else because the probability will remain quite low.
Note also that the analysis you make in the last part of the question is incorrect. It is made under the assumption that the probability of each ball to occur is identical, which you claim is invalid. If there are 11 balls that are less likely to be selected the probability that the first ball is not among those 11 is higher that $(1 - 11/36)$, etc.