Lie algebra of the group $C(3)$ has the following generators:
\begin{align} \xi^T_a &= \partial_a, & \xi^R_a &= \varepsilon_{abc} x^b \partial_c,\\ \xi^D &= r\partial_r, & \xi^S_a &= 2 x^a x^b \partial_b - r^2 \partial_a, \end{align}
where $\xi^T_a,\xi^D,\xi^R_a,\xi^S_a$ are generators of translations, dilatations, rotations and special conformal transformations respectively. However when I calculated the following commutator:
\begin{equation} [\xi^T_a,\xi^D]=\partial_a r \partial_r + r\partial_a\partial_r - r \partial_r\partial_a = \frac{x_a}{r}\partial_r \notin c(3). \end{equation}
It seems that the algebra does not close. What does it say about the algebra (did I make any mistake?) and the group structure?
You didn't say what $r$ and $\partial_r$ are supposed to be, but I assume $r = |x| =\big( \sum_a(x^a)^2\big)^{1/2}$, and $\partial_r$ is the radial coordinate vector field in spherical coordinates. Given that, the vector fields $\partial_a$ and $\partial_r$ do not commute. They are not coordinate vector fields in the same coordinate chart. If you write $r\partial_r$ in Cartesian coordinates as $r\partial_r = x^a\partial_a$, you'll fix the problem.