When crafting an item in Final Fantasy XIV, you get to see the probability it'll be a 'high quality' or HQ item. This is expressed as a probability from 1% to 100%, rounded to the nearest percent.
Naturally, people think this is biased, especially when they fail two 99% crafts in a row. But I don't know how you'd go about determing whether it is biased; the probability will in general be different on every craft. One time you craft an item you might get 85% and succeed, the next time you might get 70% and fail, even if you take the exact same steps.
My model is that that the true probability is some function $f(p)$ of the given probability; I think $f(p) = p$, but I"m not sure. My initial guess would be to do a bunch of crafts and look at at $H = \sum S_i \log(p_i) + (1-S_i) \log(1-p_i)$ (where $S_i = [\textrm{craft attempt $i$ succeeded}]$) and compare that to the distribution of $H$ you'd get for a fair RNG, but I'm not sure what that distribution would look like.
I strongly suspect you are seeing people's tendency to remember surprising events. If you are told there is a $90\%$ chance of success and succeed, you forget it easily. If you are told there is a $90\%$ chance of success and fail, you remember it. One is also more prone to remember the disappointment of failing a $90\%$ chance of failure than the surprise of succeeding a $10\%$ chance.
That said, random numbers are a lot harder than they look, and I suspect the authors haven't worked hard on this. There must be a rand() function in whatever language they program in and unless somebody on the team is nuts about this subject I would bet they use it uncritically. They need a lot of random numbers. The best (of a very small sample) discussion I know of some of the pitfalls is in Numerical Recipes, chapter 7.
I think the hypothesis we are trying to check is that the chance of success is uniformly lower (or higher) than the game claims. We need to collect a lot of unbiased data of claimed chance of success and whether the effort succeeded. A simpleminded approach is to compute the expected number of successes and compare to the actual number of successes. We can also compute the variance to see when a discrepancy is large enough to care about.
The challenge is making sure your data is accurate and unbiased. I don't know how often you try to craft an item, but it might be hard to get enough data from your own playing. If you look to other people to help, there may be a tendency to start collecting data after an unexpected failure (and including that event), which will bias the results. Ideally you would have enough data at each percent level to check whether that percentage of efforts succeed. That would take at least hundreds of thousands of attempts, probably millions. How you get that data is an exercise left to the student.