Why does it make sense to add some rates but not others?
Say tap $A$ takes $4$ minutes to fill a cup and tap $B$ takes $2$ minutes to fill a cup.
Then tap $A$ fills $\frac 1 4$ of a cup in a minute and tap $B$ fills $\frac 1 2$ of a cup in a minute. Together they fill $\frac 1 4 + \frac 1 2 = \frac 34$ of a cup in a minute and therefore they take $\frac 43$ minutes to fill a cup.
In the example above we added the rates in cups/minute and then took the inverse to get the answer in minutes/cup. Why does it make sense to add these rates and get an acceptable answer, while we can't directly add the given rates in minutes/cup. Not only does this give us a different answer since $\frac{1}{\frac{1}{a}+\frac{1}{b}}$ is not the same as $a+b$ but logically it doesn't make sense because adding two rates gives $4 \frac{\text{min}}{\text{cup}} + 2 \frac{\text{min}}{\text{cup}} = 6 \frac{\text{min}}{\text{cup}}$ (a longer time), and also doesn't make sense because even though mathematically the units are consistent, I am unsure of what the quantity represents.
The correct elegant analysis in the third paragraph of your question is the answer I would give if you hadn't.
For an extreme example of the seeming paradox, suppose the second tap is really slow and takes years to fill a cup. Then together they fill the cup in a tiny bit less than four minutes.
If you were filling multiple cups sequentially using both taps at the same time then you could fill six cups per minute. Whether you add the rates or average them as you did depends on the question you want to answer.
Read more about the harmonic mean.