Let $G$ be a group. I want to check whether the set
$$\{ a \mid a\not \in [G,G],\ a \in Z(G) \} \cup \{e\}$$
is a subgroup or not, where $[G,G]$ and $Z(G)$ are the commutator subgroup and center of the group $G$ respectively.
In a vector space $V$ we can find the complement subspace $U$ of any its subspace $U$ such that $V=W\oplus U$? Can we generalize this to my case?
This is not true.
Let $G$ be any finite group where $[G,G]\cap Z(G)$ is a proper, non-trivial subgroup of $Z(G)$. (E.g., $C_2\times D_8$.) Thus it contains at most half of the members of $Z(G)$. Your subset, therefore contains more than half of the members of $Z(G)$ (as it also includes the identity), so cannot be a proper subgroup by Lagrange's theorem.