Motivation:
Consider $\mathcal{X}=(X, +)$, where $X=\{-1, 0, 1\}$ and $+$ is standard addition. Then $\mathcal{X}$ is associative (where defined) but not closed.
NB: There is an identity element in $X$ and inverses exist in $X$ all with respect to $+$.
This example is taken from here.
The Question:
This question seems difficult to pose due to certain subtleties so, to make life easier, here's the rough idea first.
What d'you call a "magma" that's associative but not closed?
An attempt at refining the question:
What do you call the mathematical objects $\mathcal{S}=(S, T, \ast)$ for which $S$ is a set and $\ast$ is some function with domain $S\times S$ and codomain some set $T$ with $S\subset T$, such that
for all $s,t,u\in S$ we have $$s\ast (t\ast u)=(s\ast t)\ast u$$ whenever $t\ast u, s\ast t\in S$ (or $T$ if that's necessary to keep the question in spirit) and
there exist $x, y\in S$ such that $x\ast y\in T\setminus S$?
(Please disregard this attempt if it complicates the idea of the question needlessly.)
Thoughts:
I'm not sure whether naming these things is necessary. I'm interested in them out of curiosity. Whether the question even makes sense, I don't know.
Are they simply subsets of semigroups?
I made sure to say function and not binary operation above, since the latter implies closure by definition.
You might have put together two different questions due to some subtleties. As I see it, you have two main possibilities. Starting from the one you wrote explicitly:
Note that this does not raise any doubts about existance of the products as the equality holds in $T$. I'm not sure this has a name but a fair bet would be something like: " $S$ is an associative subset of the magma $T$ " or maybe " $T$ is locally associative over the subset $S$ ".
A set $S$ with a partial operation $\star: S\times S \rightarrow S$ for which, for all $s,t,u\in S$, $$ \text{ if } s\star t\text{ , }t\star u\text{ , }(s\star t)\star u \text{ , }s\star(t\star u)\in S \qquad \text{then } (s\star t)\star u = s\star(t\star u) $$ (Meaning that: if all the products involved exist, then $\star$ is associative)
This seems to be quite similar to the notion of a partial semigroup.
Using this view you could also see $S$ in definition 1. as a "partial subsemigroup" of the magma $T$ (similarly to what is done for the notion of a subgroup of a semigroup).