Let's say I have a $1:1$ ratio and a $5:1$ ratio which could also be interpreted as a $1:5$ ratio. the fact that both of these ratios are valid but one is $5$ while the other is $0.2$ is the problem, because if we plug this into the percent error formula ($|E−T|/|T|$) we get both $80\%$ and $400\%$. which would be correct? what are the rules for this? Right now I just need to figure out how to do this mathematically.
To clarify, this is with empirical formula where you have a ratio of the elements, to $1:5$ and $5:1$ would be equivalent, only because the ratio we are comparing against in this case is $1:1$, this is an edge case.
You are expecting computing errors and computing ratios to commute, which is not the case. Let us take a simple case, where you want to mix $5$ units of $A$ with $1$ unit of $B$. We will imagine you can measure $A$ perfectly, so the only error is on the quantity of $B$, which we call $\Delta B$.
One ideal ratio is $\frac BA=0.2$. If we make an error $\Delta B$ in the quantity of $B$, the ratio changes to $\frac {B+\Delta B}A$ and the error is about $\frac {\Delta B}A =\frac 15\cdot \frac{\Delta B}B$
Another ideal ratio is $\frac AB=5$ If we make an error $\Delta B$ in the quantity of $B$, the ratio changes to $\frac A{B+\Delta B}$ and the error is about $-{\Delta B}\frac A{B^2} =-5\cdot \frac{\Delta B}B$
These work as long as $\Delta B \ll B$. You need to adjust your expectations.