I'm trying to find a closed form of the generating function
$$ G(x) = \sum_{n \ge 0} r^\binom{n}{2} x^n $$
for a real number $0 < r < 1$. I found that $G(x) = 1 + xG(rx)$. Any hints where to go next?
Taking into account the answer by @zyx below, let
$$ H(x) = \sum_{n\in\mathbb{Z}} r^\binom{n}{2} x = \prod_{m\ge1} (1-r^m)(1+r^{m-1} x)(1+r^m x^{-1}), $$
where the equality of the sum and product comes from the Jacobi triple product. Note that $H(rx) = xH(x^{-1})$.
Also, using $\binom{-n}{2} = \binom{n+1}{2}$, we can obtain $\sum_{n<0} r^\binom{n}{2} x^n = xG(x^{-1})-x-1$.
Now what?
This is essentially the problem solved by Jacobi's triple product formula/identity.
http://en.wikipedia.org/wiki/Jacobi_triple_product