All the followings are $\bmod$ $p$
Let p be an odd prime. Suppose that the set $X = \{ 1,2, . . . , p-1\}$ can be written as the union of two nonempty subsets $S$ and $T$, where $S \neq T$, such that :
(1) The product of any two elements in the $S$ lies in $S$
(2) The product of any two elements in the $T$ lies in $S$
(3) The product of any element in $S$ with any element in $T$ lies in $T$.
Prove that S consists of the quadratic residues and T of the non-residues.
Working so far:
Obverse that for any $x \in X$, either $x \in S$ or $x \in T$, therefore $x x \in S$ by (1) and (2), so S consists of all square elements, i.e. $Q_p \subseteq S$ , where $Q_p$ is the set of all quadratic residues of p.
OR: Let any $x \in Q_p$, then either $x \in S$ or $x \in T$ because $S \cup T = X$, also that $xx \in Q_p$ since $\big(\frac{xx}{p}\big) = \big(\frac{x}{p}\big)\big(\frac{x}{p}\big) = 1.1 = 1$. And $xx \in S$ by (1) and (2). So $Q_p \subseteq S$.
Converse inclusion: Let any $x \in S$, then $ xx \in S $ by (1), also $ xx \in Q_p $ by definition of $Q_p$. so $S \subseteq Q_p$, therefore we have $S = Q_p$.
Fact: 1 is in $S$, not in $T$: suppose 1 $\in T$, then $1.t \in S$ for any $t \in T$ by (2). But then $1.t = t \in T$, yet its is in S, so t is in the intersection $Y = S \cap T$. This is true for any $t \in T$, so $T \subseteq Y$. Similarly, $1.s \in T$ for any $s \in S$, and that $1.s = s \in S$, so $s \in Y$ for all $s \in S$. so $S \subseteq Y$. It is obvious that $Y \subseteq S$ and $Y \subseteq T$ (since Y is the intersection of the two). Therefore we have $Y = S = T$, which is a contradiction since $S \neq T$ by hypothesis. Therefore $1 \not\in T$, so it must be in $S$ because $S \cup T = X$ and $1 \in X$.
Show $S$ and $T$ are disjoint:
Suppose there exists a $x \in X$ such that $x \in Y = S \cap T$ (Looking for building $I$ bit by bit and eventually $Y = S = T$ and hence a contradiction since $S \neq T$, similar to show 1 is not in $T$).
Let $s_i \in S-\{1\}$, $t_j \in T$. (where $s_i$ is the i^th element of $S$, similarly, $t_j$ is the j^th element of $T$. Of course rearranging the elements in order first) Then we have $xs_i \in S$ and $xt_j \in T$ using $x \in S$, and we also have $xs_i \in T$ and $xt_j \in S$ using $x \in T$. So we have $xs_i, xt_j \in Y$.
Also we have the formula: $|S| + |T| - |Y| = |X| = p-1$, not sure if it will help though.
(Continue to update)
This is as far as I get at the moment. If I can show that $S$ and $T$ are disjoint, then $T$ is just the non-residue, and we are done (or are we?). However, I am not sure how I should get through the disjoint part. I am not even sure if the above argument for $Q_p = S$ is valid.
I have seen another approach by using group theory, that shows the quotient group $X/S$ has only two elements, $S$ and $T$. Then use the fact $|X/S| = \frac{|X|}{|S|}$, ...
A details proof of this approach would be great, but the extension of my approach would be perfect. Or are these two approach essentially equivalent?
Update: Let $|S| = m$ and $|T| = n$, then since we are multiplying $x$ with $x_i$ and $t_j$, for each $2\leq i \leq m$ (assume $s_1 = 1$) and $1 \leq j \leq n$. Then we get a list of numbers in $Y = {x, xs_2, xs_2, ... xs_m, xt_1, xt_2, ... , xt_n}$. Since $s_i$ and $t_j$ are distinct within their sets, and that X is a cyclic group, then the elements in $Y$ are all distinct within $Y$, therefore $|Y| = 1 + (m - 1) + n = m + n = |S| + |T|$. So $|T| + |S| - |Y| = p-1 $ imply $ p = 1 $, contradiction.
OR: Since $x \neq 1$, and p is odd prime, x has an multiplicative inverse in either $S$ or $T$ (again, because $X = S \cup T$). So either $s_i$ or $t_j$ for some $i$ and $j$ is the inverse of $x$. Yet, this product $xs_i$ or $xt_j$ is in the intersection $Y = S \cap T$, so $1 \in Y$, in particular, $1 \in T$ with we have already established that if $1 \in T$ then we get $Y = S = T$ and hence a contradiction.
Please help and thanks in advance :)
Your proof that $S\subseteq Q_p$ is not correct; all you have shown is that every element of the form $xx$ is in $S$, which again shows that $Q_p\subseteq S$.
First show that $S\cap T = \emptyset$: suppose $x\in S\cap T$, and choose any $z\in X$. Then $zx^{-1}\in S$ or $\in T$, so that $zx^{-1}x = z\in S\cap T$ and thus $S=T=X$, a contradiction. So $S$ and $T$ are disjoint. But this means that they must have equal cardinalities, for choose any element $x\in T$; the map $f:T\to S:z\to zx$ and the map $g:S\to T:z\to zx^{-1}$ are inverses (note that $x^{-1}\in T$ as well since $xx^{-1}=1\in S$).
Since you already know that $Q_p\subseteq S$, all that remains to show is that there are exactly $\frac{p-1}{2}$ quadratic residues in $X$. Presumably you can carry this through.