I realize the title is perhaps not the most helpful. I am aware of several uses of the word "trivial," and I'm hoping that perhaps someone can provide some further insight.
1) Trivial sub-objects, i.e. if $G$ is a group, then $1, G \leq G$ are trivial subgroups of $G$.
2) Trivial solutions to a problem, i.e. $y=y'$ has a trivial solutions as a differential equation, $y=0$.
3) Trivial factors of a number or polynomial, i.e. $1,n|n$ $ \forall n \in N$
What these two meanings have in common is that they are defined in this way. Trivial subgroups or solutions are well-defined ideas. But it seems to me that many times, the word is also used in association with proofs when referring to arguments that are simple or otherwise straight forward. It seems obvious that we would say that a proof of a simple theorem like "even's squared are even" or some such would be "trivial," but a quick look at the top unanswered questions right now reveals many problems that are not trivial at all. Why is this so? Is triviality just defined by common consensus that a problem is hard? If the issue is really so subjective, is the claim "all mathematics is trivial" just as valid as any other claim about it (even if this seems a bit ridiculous)?
I feel inclined to point out that I don't believe that mathematics is a trivial subject, I'm just trying to understand what this common expression really means. It seems sometimes it is quite easy for the reader, and sometimes quite difficult!
There are two meanings of the word "trivial" in mathematics. The first, as you rightly pointed out, is strictly defined and appears in most mathematical fields. By strictly defined, I mean that if you say "the trivial solution to the ODE $y'=g(x) y$", I know with certainty that you mean the solution $y\equiv 0$. The same goes for trivial subgroups, trivial topological spaces, trivial vector subspaces and so on. In each case, the word trivial has a well defined meaning and is in no way ambiguous.
The second meaning is more tricky. The second meaning of the word trivial can best be replaced with "very simple". For example, the proof that the number $7$ is a prime number can be considered trivial.
It is clear that this definition of the word is much more subjective than the first. For example, a $10$ year old child will find it very hard to understand that the cardinality of $[0,1]$ is the same as the cardinality of $\mathbb R$, while on the other hand, a seasoned set theoretician will never bother to actually spell the proof of this fact in his paper as him and anyone reading his paper will consider it to be trivial. It is also possible (and happens often) that certain theorems and facts which seem trivial to one mathematician will be confusing to another, especially if the two come from different fields.
My bottom line would be this: you are correct. The definition of the word "trivial" is a matter of consensus, and that consensus can change even among mathematicians. It is important to not that you may claim something is trivial only if a vast majority of mathematicians in your field also consider it trivial.
The last issue you raised was "is all mathematics trivial", to which I cannot give a good answer. The thing is that each and every mathematical proof can be decomposed into it's simplest components, each of which is trivial (if this cannot be done, then it is not a mathematical proof). But does this mean that the proof is necessarily trivial? In my opinion, no. Not at all. Trivial does not mean that it can be done, it means it can be done by anyone with some background very easily.