From Interpretability Methods to Interpretable Models

Read Julien's PhD thesis: PDF.

Abstract

Deep learning models are increasingly used to make consequential decisions, yet remain largely opaque to the people who rely on them. Over the past decade, explainable AI (XAI) has developed a mature toolbox for generating explanations, but that toolbox has mostly been turned inward: it has been used primarily to compare explanation methods against each other rather than to ask whether the resulting models are actually understandable. This thesis redirects the focus of XAI from evaluating explanations to evaluating the models those explanations are intended to illuminate. It develops a framework for measuring and comparing the interpretability of vision models, and for causally testing whether that interpretability can be deliberately shaped. The framework is built across three chapters, which address three fundamental questions: what representational basis should interpretability be measured over; how can human understanding of models be measured and compared across today's vision foundation models; and can interpretability be improved through targeted interventions rather than simply observed?

The first chapter establishes the appropriate representational basis for interpretability. Individual neurons often respond to several unrelated visual patterns, making them an unreliable unit of explanation. Dictionary learning has been proposed as an alternative, but its success has largely been evaluated using technical criteria rather than the human judgments that determine whether a representation is genuinely interpretable. This chapter closes that gap through three large-scale psychophysics experiments (N=481) across two convolutional networks. Dictionary-based features are consistently more interpretable than neurons, with the advantage increasing in deeper layers. Moreover, the choice of representational basis itself can determine whether differences between models are detected at all.

Building on this foundation, the second chapter turns to the problem of measurement. Existing protocols for comparing model interpretability rely on forced-choice tasks that allow participants to succeed through elimination rather than genuine understanding, making fair cross-model comparisons difficult. This chapter introduces two behavioral protocols, localizability and semanticity, that evaluate interpretation directly and place models on a common, chance-corrected scale. Applied to six vision transformers, including four foundation models (DINOv2, DINOv3, CLIP, SigLIP), and based on more than 15,000 human responses, the protocols reveal that foundation models are consistently less interpretable than their supervised counterpart. This gap is explained neither by model scale nor by downstream task performance. Instead, feature locality and alignment with human semantic judgments emerge as the strongest predictors of interpretability.

The third chapter examines whether these predictive factors can be turned into causal levers for interpretability. Five vision backbones are LoRA-finetuned toward human perceptual judgments and the resulting aligned models are evaluated using the same behavioral measures developed in Chapter 4. The results show that alignment is an unreliable lever for improving interpretability: its effect is positive across the three vision-only backbones, reaching significance only for DINOv2, and reverses for the two vision-language models. Even when alignment improves interpretability, the underlying mechanism remains inconsistent and only partially understood. Thus, while alignment can serve as a lever for improving interpretability, it does not provide a reliable or universally applicable solution, and the reasons it succeeds for some models while failing for others remain unresolved.

These contributions provide the first behaviorally grounded account of how interpretable current vision foundation models are, why they differ, and how far their interpretability can be deliberately improved. More broadly, this thesis argues that progress in explainable AI requires moving beyond the evaluation of explanations alone and toward the systematic study of whether models themselves become understandable to humans. The framework developed here provides a foundation for that shift, and points toward future models whose representations---and ultimately whose decisions---can be meaningfully understood by the people who depend on them.

Main publications discussed in the thesis

Colin, J., Oliver, N., & Serre, T (2026). From Interpretability Methods to Interpretable Models. ECCV Workshop on Explainable Computer Vision. ECCV

Colin, J., Goetschalckx, L., Fel, T., Boutin, V., Serre, T., & Oliver, N. (2026). Choosing the Right Basis for Interpretability: Psychophysical Comparison Between Neuron-based and Dictionary-based Representation. Transactions on Machine Learning Research. TMLR

Colin, J., Oliver, N., & Serre, T (2026). Can Human Perceptual Similarity Alignment Improve Interpretability in Visual Representations?. Conference on Cognitive Computational Neuroscience (CCN)[https://2026.ccneuro.org/poster/?id=7cDVJL4Kgo]

Colin, J., Goetschalckx, L., Oliver, N., & Serre. T. (2026). Capability $\neq$ Interpretability: Human Interpretability of Vision Foundation Models. arXiv

Colin, J., Oliver, N., & Serre, T (2026). Does Human-Alignment Benefit Interpretability?. ICLR Workshop on Representational Alignment ($Re^4$-Align) ICLR