If a set is represented by curly bracket $\{\ \ \}$
Then why is a closed set represented by $[\ \ ]$ and open set represented by $(\ \ )$
If a closed set is represented by $[\ \ ]$ , then is a vector
$$x = \begin{bmatrix} x_1 \\ x_2 \end{bmatrix}$$
also a set? E.g., a set containing $x_1, x_2$.
The notation you are referring to is pretty much specific to intervals in $\Bbb R$ with the usual topology. This is a pretty specific setting. Open and closed sets are concepts in topology, and for a general topological space, there is no notion of closed or open intervals, or even intervals at all.
In $\Bbb R$ with the usual topology, $[a,b]$ is just shorthand for $\{x\in\Bbb R:a\le x\le b\}$. Similarly, we write $$(a,b)=\{x\in\Bbb R:a<x<b\},$$ but the exact same notation $(a,b)$ in another context means a single point in $\Bbb R^2$.
In the example you gave, the square brackets in $$\begin{bmatrix}{x_1\\x_2}\end{bmatrix}$$ only indicate that the object is a vector. Formally, this vector would probably be considered a pair $(x_1,x_2)\in\Bbb R^2$, which is not to be confused with the set containing that pair, i.e., $\{(x_1,x_2)\}$. If you want to talk about open and closed subsets of $\Bbb R^2$, you have to know precisely what that means first, since you can no longer even refer to intervals.
Plenty of authors use rounded parenthesis for vectors and would write the exact same vector as $$\begin{pmatrix}{x_1\\x_2}\end{pmatrix}.$$ In combinatorics, we typically interpret the above notation instead as a binomial coefficient: $$\begin{pmatrix}{m\\n}\end{pmatrix}=\frac{m!}{n!(m-n)!}.$$ My point is that it's okay for notation to be reused as long as the precise meaning is clear from context.
Returning to my first sentence, I want to emphasize that "open" and "closed" are concepts in topology, not set theory. Given an arbitrary set, there are no open or closed sets at all until you define a topology. Even when you have a topological space, the open and closed sets can be quite weird, and you can't always rely on open and closed intervals of $\Bbb R$ for intuition.