From the Open Logic Project book 2.2, Philosophical reflections (Set theory):
Third: when we “identify” relations with sets, we said that we would allow ourselves to write Rxy for ⟨x, y⟩ ∈ R. This is fine, provided that the membership relation, “∈”, is treated as a predicate. But if we think that “∈” stands for a certain kind of set, then the expression “⟨x, y⟩ ∈ R” just consists of three singular terms which stand for sets: “⟨x, y⟩”, “∈”, and “R”. And such a list of names is no more capable of expressing a proposition than the nonsense string: “the cup penholder the table”. Again, even if some relations can be treated as sets, the relation of set-membership must be a special case. (This rolls together a simple version of Frege’s concept horse paradox, and a famous objection that Wittgenstein once raised against Russell.)
I don't understand the paragraph. Why is the list of names no more capable of expressing a proposition? What are Frege's concept horse paradox and Wittgenstein's objection, and how are they related to all this?
A list of names, "Romeo, Juliet", or "Socrates, Plato, Aristotle", or “the cup penholder the table” is just that, a list of names, and doesn't express a proposition, can't say anything true or false. A list of names is just a plural designator: so the list "Catherine of Aragon, Anne Boleyn, Jane Seymour, Anne of Cleves, Catherine Howard, Catherine Parr" denotes the same six women as "the wives of Henry VIII". But it doesn't say anything about them, doesn't express a proposition which can be true or false.
Contrast e.g. "Romeo dreams" which does express a proposition. Which it couldn't if "dreams" really were another name on a par with "Juliet".
No, "dreams" is a one-place predicate. Its role here is not to name something but to attribute a property. And a list of predicates like "dreams swims coughs" doesn't express a proposition any more than a list of names does.
Note too, "Romeo, the property of dreaming" is also a list, not a sentence expressing a proposition. It lists the boy and the property -- but it doesn't say anything about them: in particular, it doesn't say that the boy actually has the property.
In type theoretic terms, if "Romeo" has the type $n$, "dreams" has the different type $n \to s$ (or $s/n$ if you prefer). And applying an expression of type $n \to s$ (a predicate) to an expression of type $n$ (a name) gives us a sentential expression of type $s$ (a sentence which can express a proposition).
Equally, "Romeo, the set of boys" is another list of names, and again doesn't express a proposition. However, "Romeo $\in$ the set of boys" (or "$r \in B$", say) by contrast does express a proposition. Which again it wouldn't if "$\in$" were just another name of type $n$. Rather, "$\in$" is a two-place predicate. Its role here is not to name some further thing in addition to Romeo and the set of boys -- its role to say that the first thing is a member of the second thing.
"$\in$" is an expression of type $n \to n \to s$ (or $s/2n$ if you prefer) which can combine with two names to form a sentence. But if "$\in$" is a predicate and not a name, it isn't the name of a set in particular.
Don't worry too much here about the details of the references to Frege and Wittgenstein: the point that they were both making is the same one -- that there is a deep categorial difference between names and predicates.
Aside: Frege has a metaphor which you might find helpful: predicates (expressions whose semantic values are properties or relations) are "unsaturated" -- with gaps waiting waiting to be completed with names (or other suitable expressions) to form a complete sentence. In fact, we would do better represent the predicate in "Romeo dreams" by "-- dreams" to indicate it is waiting to be completed with a name, and represent the membership predicate by $\ldots \in --$ to show it it has two gaps waiting to be filled.
Finally, on that remark "Again, even if some relations can be treated as sets, the relation of set-membership must be a special case." Well, I wouldn't want to treat any relations as sets -- that's a type confusion, say I. But we can for certain purposes represent or implement or model the relation of loving (for example) by the set $L$ of ordered pairs $\langle m, n\rangle$ such that $m$ loves $n$. So it will then be true that Romeo loves Juliet if and only if $\langle r, j\rangle \in L$. But note again, that although "$\langle r, j\rangle$" is a name (of an ordered pair), and "$L$" is a name (of a set, the extension of the relation of loving), it can't be that "$\in$" here is also a name -- it has to be a two-place predicate attributing a relation if the whole expression "$\langle r, j\rangle \in L$" is to express a proposition that can be true or false.