I'm looking at equations for neural networks and backpropagation and I see this symbol in the equations, ⊙. I thought matrix multiplication of neural networks always involved matrices that matched dimensions on both sides, such as... [3, 3]@[3, 2]. (This is what is happening in the animated gif).
This is not element wise Hadamard multiplication ⊙ is it?
That animation shows standard matrix multiplication, but they are showing it in a somewhat strange way. It makes sense, but if you don't already know what's going on, it's nearly impossible to keep up with what it's doing.
Hadamard product is the "easiest" matrix product. It works exactly the way you would think matrix product ought to work before you're introduced to the subject: You just muliply corresponding elements together, with no pesky addition making the formulas messy. This does require that both operands have exactly the same shape, and the resulting product will also have the same shape.
As for when to use each method, that is really difficult to answer thoroughly. But if you want to be lazy, then one shortcut could be to look at what combination of matrix shapes makes sense. If you want your product to be a $3\times 3$ matrix with a $3\times 2$ matrix, then you can't do the Hadamard product, so standard product could be a good candidate. If you want to multiply a $4\times 2$ matrix with a $4\times 2$ matrix, then you can't do the standard product, so Hadamard might be the right one.
But the only good, general way to tell, is to actually analyze your situation, and write out the operation you want to do. And only after the fact can you take a step back, look at it and see whether you recognize it as the standard product, or the Hadamard product, or the Kronecker product, or something else.