With some category/topos theory we can now put infinitesimals on a rigorous ground, as in Bell's A Primer of Infinitesimal Analysis, where the author introduces $\epsilon$ satisfying \begin{equation} \epsilon\ne 0, \epsilon^2=0. \end{equation}
However, he also points out that this version of infinitesimal is not compatible with the law of excluded middle.
Meanwhile, the author seems convinced that this $\epsilon$ is the infinitesimal in the eyes of Newton and Leibniz among many others, when they were attacking problems like instantaneous speed and area under a curve.
I wonder whether this is true. I know people like Newton and Leibniz did not use limiting argument. But this does not mean they think of infinitesimals as nilsquare elements as described by Bell, because there are still other models of infinitesimals available.
Thanks very much.
The reals admit to axiomatization that allows for infinitesimals. Look at http://en.wikipedia.org/wiki/Hyperreal_number
Edit: Excuse me. I misread the question. No one can say with certainty. The ancients must have thought of infinitesimals as "very small" or "infinitely small" change, as in the differential change "dx." Newton and Leibniz did not have a formal definition of the reals and were not aware of their completeness. They had superb intuition. I doubt anyone will give you a more satisfactory explanation.
p.s. This question feels a lot like the high school:
What did Dumas mean when he wrote such and such in his "blah-blah?"