Softmax Linear Units (SoLU)

Elhage, Hume, Olsson et al. (Anthropic) (2022)

Read paper

Tags: foundations, architecture, anthropic

Abstract

We investigate SoLU, an activation function designed to encourage monosemantic neurons in transformers, as a step toward making neural networks more interpretable.