Toy models of superposition
Why don't we get grandmother neurons - neurons corresponding to a feature?
The paper uses small ReLUs with sparse input features to see how superposition occurs. This is when the number of represented features exceeds the number of dimensions.