I want to find the growth rate of the function $$f(n)=\sum_{i=1}^\infty (1-1/n)^{i(i-1)/2}$$
Since $(1+x)^k\approx 1+kx$ for $|x|\ll1$, I modified $f(n)$ to $$ f(n) \approx \sum_{i=1}^\infty \left[1-\frac{i(i-1)}{2n}\right] $$ for $n$ large, but this diverges. How can I find the correct growth rate? I strongly conjecture it will be $O(\sqrt n)$.
Since the function $g_n(x)=(1-1/n)^{x(x-1)/2}$ is monotone decreasing for $x\ge1$, we have $$ \int_1^\infty g_n(x)dx\le f(n)\le \int_1^\infty g_n(x)dx+1-1/n. $$ (Hint: Compare the integral with its Riemann sums, where the integer points are takes as nodes.) Now $$ \int_1^\infty g_n(x)dx=\int_1^\infty e^{\ln(1-1/n)x(x-1)/2}dx =\frac1{\sqrt{-\ln(1-1/n)}} \int_1^\infty e^{-y^2/2+y/(2\sqrt{-\ln(1-1/n)})}dy. $$ The last integral is bounded above and below by positive constants independent of $n$, and it remains to note that $-\ln(1-1/n)\approx 1/n$ and hence $$\frac1{\sqrt{-\ln(1-1/n)}}\approx\sqrt n$$ for large $n$. Thus, your guess is correct.