I know that a probability measurable space $X$ comes with a measure $\mu$ s.t $\mu(X)=1$ and $\mu(B) \in (0,1); \forall B\in \mathfrak B$.
Now let $X$ be compact metric probability measurable space with measure $\mu$.
Now an atom of a probability space is a measurable set $A$ with positive measure $P(A)$ and the property that for each measurable subset $B\subseteq A$, either $P(B)=0$ or $P(B)=P(A)$. The probability space is purely atomic if every measurable set with positive measure contains an atom. If $A$ is an atom, it will still be an atom in the completion of the probability space. So yes, a complete probability space can be atomic. Every countable probability space is purely atomic.
To prove that if $\mu$ is purely atomic then $\mu=\sum_{i=1}^\infty > p_i \delta_{x_i}$ s.t $\sum_{i=1}^\infty p_i=1$ and $p_i$'s are the only weight what an atom can get (i.e WLOG if we assume that $p_1 \geq p_2 \geq \cdots \geq$ then the highest weight an atom can get is $p_1$).
Let $A$ be an atom. We can write $X$ as a finite union of closed balls of diameter $1/2$. Intersect these sets with $A$ and conclude that one of the intersections, say $A_1$ has $\mu (A_1)=\mu(A)$. $A_1$ is the intersection of $A$ with a closed ball of diameter $1/2$. Write this ball as a union of closed balls of diameter $1/3$ and conclude that there is a set $A_2 \subset A$ such that $\mu (A_2)=\mu (A)$ and $A_2$ is the intersection of $A_1$ with a closed ball of diameter $1/3$, and so on. Inductively construct sets $A_n$ and conclude that there is a single point, say $x$ in the intersection of these sets. Now $\mu (\{x\})=\lim \mu(A_n)=\mu(A)$. It follows that every atom differs from a singleton set only by a set of measure $0$. Now $\{x:\mu(\{x\})>0\}$ is at most countable. Call this countable set $\{x_1,x_2,...\}$ and let $p_k=\mu(\{x_k\})$. If $E$ is any measurable set and $\mu (E \setminus \{x_k:k\geq 1\}) >0$ then $E\setminus \{x_k:k\geq 1\}$ would contain an atom, hence a singleton of positive measure and this is a contradiction. Now can you verify that $\mu (E)=\sum_{i=1}^{\infty} p_i \delta_{x_i} (E)$?