SAEfarer: exploring text classification models with sparse autoencoders
Leveraging SAEs to analyze the behavior of text classification LMs.
As language models (LMs) rise in prominence, there is interest in making them more transparent in order to better understand and steer their internal behavior. Recent work in interpreting LMs has focused on using sparse autoencoders (SAEs) to break down neuron activations at a given layer in the LM into human-understandable features, where each feature represents a single concept that the model knows. In this paper, we present initial work on leveraging SAEs to analyze the behavior of text classification LMs. We present techniques for exploring the relationships between the features extracted by the SAE and the model’s predictions and errors. We integrate these techniques into SAEfarer, an open-source prototype visual analytics tool for analyzing text classification models.
Latest publications
RECAP: Regression evaluation for continual adaptation of prompts
A benchmark that measures continual-learning phenomena at the constraint level for prompt-level adaptation methods.
EMNLPMEMGUARD: Preventing memory contamination in long-term memory-augmented large language models
A type-aware memory framework that preserves functional memory boundaries during memory construction and retrieval.
EMNLPA history-aware visually grounded critic for computer use agents
A test time intervention framework for long-horizon GUI agents.
EMNLP