So I am supposed to:
Consider a decimal machine in which floating point numbers are represented with a precision of 12 decimal places and compute the relative error for numbers (assume numbers are rounded correctly).
[ | x - fl(x) | ] / |x| <-- relative error equation
Ex) 2.3456 – 0.00000456789011234
If you normally do the subtraction, fl(x) = 2.34559543210988766
My problem is I'm not sure if the fl(x) should be 2.345595432109 (which is the above computation rounded to the 12th decimal place).
OR
That fl(x) = 2.3456 – 0.000004567890 = 2.34559543211 (numbers are rounded BEFORE the subtraction).
The relative error itself is simple since you just plug and chug, but I'm not sure which number would be correct for fl(x) or if either of the two methods I stated above are even correct.