Given $(G,•)$ as a Group with finite set $G$, operator •.
Define: subset $S \subset G$ is called the core of the Group if and only if
$$ \{ x•y ~|~ x \in S, y \in S \} = G \setminus S$$
Conjecture 1: if $G$ has at least two elements, core exists.
Conjecture 2: if core exists, any two cores have the same cardinality.
Conjecture 2.1: given prime $p \ge 3$, for $G = \{1,2,\ldots,p-1\}$ and operator • is 'multiplication modulo p', then any core $S \subset G$ has cardinality $(p-1)/2$.
Example: $p = 7$, $S = \{3,5,6\}$
$1 \equiv 3*5 \equiv 6*6\pmod{7}$
$2 \equiv 3*3 \equiv 5*6\pmod{7}$
$4 \equiv 3*6 \equiv 5*5 \pmod{7}$
Question: anyone can find any (related) reference of the definition of core?
Note. Core already has a meaning in Group Theory: given a group $G$, the “core of a subgroup $H$” is the largest normal subgroup of $G$ contained in $H$. Since your set $S$ can never contain the identity, it can never be a subgroup, so this meaning of “core” is antithetical to the standard meaning. Please consider changing the name.
Conjecture 1 is false. Consider the cyclic group of order $3$, generated by $x$. If $S$ were a “core” of $G$, then it cannot contain $e$ (since $e\bullet e = e\notin G-S$). Now, if $x\in S$, then $x^2=x\bullet x\notin S$; and if $x^2\in S$, then $x=x^2\bullet x^2\notin S$. Thus, $S$ would have to be either empty (in which case it cannot satisfy your condition) or else $S$ contains exactly one element, in which case $\{s\bullet t\mid s,t\in S\}$ contains exactly one element, and cannot be equal to $G-S$.
It is true that the quadratic nonresidues form a set as given. More generally, if $G$ has a subgroup $H$ of index $2$, then $G-H$ works as a set $S$, since $H$ is normal and $G/H$ is cyclic of order $2$; thus the product of two elements not in $H$ is always in $H$, and every element of $H$ can be expressed that way: take $h\in H$, $x\notin H$; then $x^{-1}(xh) = h$ is a product of two things in $G-H$.
Conjecture 2 is also false. Consider $G=S_3$. As noted above, if we take $H=\{e, (1,2,3), (1,3,2)\}$ which is a subgroup of index $2$, then $S=G-H = \{(1,2), (1,3), (2,3)\}$ will satisfy $SS=G-H$. However, so does $S’=\{(1,2), (1,2,3)\}$: indeed, we obtain the other four elements as the products (I compose permutations right-to-left): $$\begin{align*} (1,2)(1,2) &= e\\ (1,2)(1,2,3) &= (2,3)\\ (1,2,3)(1,2) &= (1,3)\\ (1,2,3)(1,2,3) &= (1,3,2). \end{align*}$$ That is, $(S’)(S’) = G-S’$. However, $S$ has three elements and $S’$ has two.