In what sense word-embedding in NLP is embedding in mathematics?

48 Views Asked by At

In Natural language processing, there is a method called "word embedding", which is defined as a map from word space to some larger real space. https://en.wikipedia.org/wiki/Word_embedding

For example, borrowing the example from this tutoraial https://www.tensorflow.org/text/guide/word_embeddings, we can consider word-embedding to $\mathbb{R}^4$ such as cat may be expressed a (1.2, -0.1, 4.3, 3.2) and mat may be expressed as a (0.4, 2.5, -0.9, 3.2). As the example shows, is seems that word embedding a map from a discrete finite set $S$ of words (e.g. $S = \{cat, mat, on, table, ...\}$) to real space, that is $S \to \mathbb{R}^n$, where $n$ is some user-specified value.

My question is how this "embedding" can be related to mathematical concept of embedding? https://en.wikipedia.org/wiki/Embedding Or, is it just a misuse of mathematical term by a machine learning engineer?

As far as I know, embedding can be defined between two-smooth manifold, and in this case $S$ is not smooth. This is confusing me. However, I understand that the embedding map here is injective, which makes sense when comparing to the embedding between manifold.