I'm aware that many differently looking definitions exist for an algebra over a commutative ring $A$. For me, the most natural definition of an algebra over a commutative ring $A$ consists of a tuple $(B, f)$ where $B$ is a ring (not necessarily commutative) and $f: A \to B$ is a ring homomorphism satisfying $f(A) \subseteq Z(B)$, i.e. $bf(a) = f(a)b$ for all $a \in A, b \in B$.
Some authors seem to require that the homomorphism $f$ is also injective, such that $A$ can be embedded into $B$. I can't seem to find whether this assumption is common.
In my undergraduate thesis I want to use that, for an ideal $I$ of a commutative ring $A$ and some $a \in A$ fixed, there exists a unique $A$-algebra homomorphism $$ \phi : A[X] \to A/I $$ which maps $X$ to $a + I$. For this I need to be able to consider $A/I$ as an $A$-algebra, which I only can when I drop the injectivity of $f$. Is this clear to a mathematical audience, without having to define what I see as an algebra in my paper?
Every ring can be considered as a $\mathbb{Z}$-algebra and, of course, the (unique) homomorphism is not necessarily injective.
It would be indeed quite strange that a quotient of an $A$-algebra is not necessarily an $A$-algebra, which is exactly the example you make.
Of course, if $A$ is a field, then the homomorphism is injective, provided the algebra is not the trivial ring.