I'm working with experimental numbers (directions in reciprocal space). That unfortunately means that the pure algorithms for GCD are not working. The standard deviation for numbers can be as high as 1%.
For example a vector [0.373, 0.007, -0.625] should be express as [3,0,-5] (numbers should be smaller than 10).
I tried to play with rounding but I couldn't find any algorithm that always works. It was relative easy to remove potential zero, but the best I could get for above one was [16,0,-27]
Could anyone point me to right direction?
Hm, if I'm following correctly, you want to be able to rescale your vectors freely, then find a vector of integers below a certain amount that it's "closest" to. I suppose that what "closest" means is somewhat subjective, but what I'd want is the one that is at the closest angle. Here's what I'd do:
$$(n_1, n_2, n_3)$$
with $n_i$ an integer between -10, 10.
Normalize all of the reference numbers. This does mean we have to throw out the zero vector.
Normalize an experimental number (this step isn't strictly necessary).
Find the reference number on the normalized list that has the largest dot product with the normalized experimental number. Choose this one.
I'm not really sure how to deal with the zero vector, though. How can you ever be close to the zero vector in a projective space?