In a recent article at Distill (link) about visualizing internal representation of convolutional neural networks, there is the following passage (bold is mine):
If neurons are not the right way to understand neural nets, what is? In real life, combinations of neurons work together to represent images in neural networks. Individual neurons are the basis directions of activation space, and it is not clear that these should be any more special than any other direction.
Szegedy et al.[11] found that random directions seem just as meaningful as the basis directions. More recently Bau, Zhou et al.[12] found basis directions to be interpretable more often than random directions. Our experience is broadly consistent with both results; we find that random directions often seem interpretable, but at a lower rate than basis directions.
I feel like they are talking about linear algebra representations, but struggle to understand how one neuron can represent a basis vector.
So at this point I have 2 main questions:
- A neuron has only a scalar output, so how that can be a basic direction?
- What is an activation space and how to intuitively think about it?
I feel like understanding these can really broaden my intuition about internal geometry of neural nets. Can someone please help by explaining or point me in the direction of understanding internal processes of neural nets from the linear algebra point of view?