Let's say I have 20 people who rated a service, the average rating for them was 4/5. If a new person did the rating for 3/5, what's the new average of rating?
Note the answer doesn't have to be exact, I am using the calculation for rating component, so I am going to round the result as I will have only integers.
Assuming that the true average rating (i.e. without any rounding) was exactly 4, then the total rating for all 20 people is $4 \times 20 = 80$. So the total rating after adding the 21st rating in is $80 + 3 = 83$, so the average rating across 21 people is $83/21 \approx 3.95$. If you don't store either an accurate value for the average, or the total of all ratings, you're going to have trouble adjusting the rating as new ones come in since rounding will tend to push everything back to the previous value.
EDIT for example:
Let's suppose you already have 100 ratings, with a total of 374, so the average is 3.74. Then if the 101st user gives a rating of ...
1, the new average will be $375/101 = 3.71$
2, the new average will be $376/101 = 3.72$
3, the new average will be $377/101 = 3.73$ 4, the new average will be $378/101 = 3.74$ 5, the new average will be $379/101 = 3.75$
If you just store that value rounded to the nearest whole number, you'll say all of those are 4. Then as new ratings come in, you'll probably keep rounding the result to 4 for as long as you like - even if a million people all give a rating of 1, if you update after every rating you'll never see the value shift. Storing to 2 decimal places means that the same problem will happen once you reach a few thousand ratings.