Also, I've heard that initially calculus was based upon infinitesimals but then it's foundations were changed from that which led to the foundations of modern theory of analysis. Does it corresponds to the lack of rigour in infinitesimal, if any, or something?
What is non-standard analysis? Is it related to the theme question?
On
The procedures of Leibnizian calculus with infinitesimals find faithful proxies in the procedures of Robinson's framework for calculus and analysis with infinitesimals, putting to rest centuries of anti-infinitesimal vitriol from Bishop Berkeley to Georg Cantor. Berkeley and Cantor thought infinitesimals were contradictory. Both published articles attempting to "prove" this. Both were wrong, as demonstrated in recent literature. See this site for a wide variety of published articles on the topic by a group of over 25 authors.
Yes, Leibniz used something one could call static infinitesimals, in modern terms one can replicate some of that with (truncated) power series arithmetic. Newton used a more dynamic idea of infinitesimals, which led to criticisms about ghost of vanishing quantities...
The use of infinitesimals in that time was more an art than a science, leading to many mistakes and misunderstandings.
Non-standard analysis tries to capture some of the ease and directness of infinitesimals, which is demonstrated by the multiple approaches (Keiser, Diener/Diener, Laugwitz) for introductory analysis courses, i.e., real analysis, metric topology. But for more advanced topics like functional analysis NSA is quite cumbersome to formulate, as one has at every step to distinguish the standard and non-standard entities.